SGLang is a fast serving framework for large language models and vision language models.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

A high-throughput and memory-efficient inference and serving engine for LLMs

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search…

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

CUDA Templates for Linear Algebra Subroutines

FlashInfer: Kernel Library for LLM Serving

DeepEP: an efficient expert-parallel communication library

The official repository for ERNIE 4.5 and ERNIEKit – its industrial-grade development toolkit based on PaddlePaddle.

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

FlashMLA: Efficient MLA decoding kernels

Perplexity GPU Kernels

QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.

A bidirectional pipeline parallelism algorithm for computation-communication overlap in V3/R1 training.

Dynamic Memory Management for Serving LLMs without PagedAttention

flash attention tutorial written in python, triton, cuda, cutlass

Fast and memory-efficient exact attention

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

Universal LLM Deployment Engine with ML Compilation

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

A throughput-oriented high-performance serving framework for LLMs

Fast Hadamard transform in CUDA, with a PyTorch interface

how to optimize some algorithm in cuda.

Stable Diffusion web UI