A high-throughput and memory-efficient inference and serving engine for LLMs

PyTorch native quantization and sparsity for training and inference

Deep learning at the speed of light.

Memray is a memory profiler for Python

Run compilers interactively from your web browser and interact with the assembly

Content addressable storage with excellent search

Implementation for FP8/INT8 Rollout for RL training without performence drop.

Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

FlashInfer: Kernel Library for LLM Serving

ZeroMQ core engine in C++, implements ZMTP/3.1

Easiest and laziest way for building multi-agent LLMs applications.

State-of-the-art TTS model under 25MB 😻

CUDA on non-NVIDIA GPUs

llm-d is a Kubernetes-native high-performance distributed LLM inference framework

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

The Modular Platform (includes MAX & Mojo)

A TTS model capable of generating ultra-realistic dialogue in one pass.

Distribute and run LLMs with a single file.

[CVPR 2025] Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass

Python pdb for multiple processes

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

A Triton-only attention backend for vLLM

A guidance language for controlling large language models.

The NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications.