🤗CacheDiT: A Training-free and Easy-to-use Cache Acceleration Toolbox for Diffusion Transformers

SpargeAttention: A training-free sparse attention that can accelerate any model inference.

📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

[NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an …

A high-throughput and memory-efficient inference and serving engine for LLMs

🛠 A lite C++ toolkit: contains 100+ Awesome AI models, support MNN, NCNN, TNN, ONNXRuntime and TensorRT. 🎉🎉

📚FFPA(Split-D): Yet another Faster Flash Prefill Attention with O(1) GPU SRAM complexity for headdim > 256, ~2x↑🎉vs SDPA EA.

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism, etc. 🎉🎉

⚡️Write HGEMM from scratch using Tensor Cores with WMMA, MMA and CuTe API, Achieve Peak⚡️ Performance.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

Code and data for the Chain-of-Draft (CoD) paper

FlashMLA: Efficient MLA Decoding Kernel for Hopper GPUs

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

DeepSeek 系列工作解读、扩展和复现。

Context parallel attention that accelerates DiT model inference with dynamic caching

[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

CUDA Templates for Linear Algebra Subroutines

Fast and memory-efficient exact attention

LMDeploy is a toolkit for compressing, deploying, and serving LLM

text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.

Development repository for the Triton language and compiler

本项目旨在分享大模型相关技术原理以及实战经验。

NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applicat…

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization and sparsity. It compresses deep learning models for downstream deployment frame…