Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

CUTLASS and CuTe Examples

flash attention tutorial written in python, triton, cuda, cutlass

coding CUDA everyday!

Efficient Top-K implementation on the GPU

OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient.

A Quirky Assortment of CuTe Kernels

Nano vLLM

A lightweight design for computation-communication overlap.

Learning CUDA

Effective transpose on Hopper GPU

Implement Flash Attention using Cute.

Ring attention implementation with flash attention

Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.

从零实现一个 llama3 中文版

Perplexity GPU Kernels

oneAPI Math Library (oneMath)

High-performance GEMM implementation optimized for NVIDIA H100 GPUs, leveraging Hopper architecture's TMA, WGMMA, and Thread Block Clusters for near-peak theoretical performance.

Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

Fastest kernels written from scratch

📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.🎉

Applied AI experiments and examples for PyTorch