Kubernetes AI Toolchain Operator

llm-d enables high-performance distributed LLM inference on Kubernetes

Production-Grade Container Scheduling and Management

Examples and guides for using the Kaito API

Gateway API Inference Extension

☁️♮🏛 This repo contains several documents related to the operation of the CNCF. File non-technical issues related to CNCF here.

This project captures work in progress, and completed work for the Azure Core Container Upstream team

helm charts for deploying models with llm-d

Helm charts for llm-d

Agent Baker is aiming to provide a centralized, portable k8s agent node provisioning lib as well as rich support on different OS image with optimized k8s binaries.

Test infrastructure for the Kubernetes project.

A high-throughput and memory-efficient inference and serving engine for LLMs

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

📜Fork for tracking CNCF projects

Cost-efficient and pluggable Infrastructure components for GenAI inference

Setup and configure nodes to support GPUs on k8s, with a focus on AKS nodes. This repo contains steps to build a container image with the Nvidia driver, and dependencies for integration.

LeaderWorkerSet: An API for deploying a group of pods as a unit of replication

JobSet: a k8s native API for distributed ML training and HPC workloads

A toolkit to run Ray applications on Kubernetes

Gatekeeper - Policy Controller for Kubernetes