RWKV (pronounced RWaKuV) is an RNN with GPT-level large language model (LLM) performance that can be trained directly like a GPT Transformer (parallelizable).

RWKV combines the best features of RNN and Transformer: excellent performance, constant memory usage, constant inference generation speed, "infinite" context length, and free sentence embeddings. It is also 100% free of self-attention mechanisms.

The RWKV project was initially proposed by Bo Peng (Blink_DL), and as the project gained attention, it gradually developed into an open-source community.

On September 20, 2023, the RWKV open-source project officially joined the Linux Foundation. Today, the RWKV project is an open-source non-profit organization under the Linux Foundation, with some computing power previously supported by sponsors.

• Discord Forum
• HF Gradio-1 | RWKV7-G1 2.9B
• HF Gradio-2 | RWKV7-G1

RWKV-7 (Goose) is the latest version of the RWKV architecture. The paper was co-authored by Bo Peng and the RWKV community, published on March 18, 2025.

• RWKV-7 Paper: "RWKV-7 Goose with Expressive Dynamic State Evolution"
• Paper Link: arXiv:2503.14456

RWKV-7 adopts Dynamic State Evolution, surpassing the fundamental limitations of the TC0 expressive power of the attention/linear attention paradigm.

::: details Click to view RWKV-7 Architecture Diagram :::

RWKV 5/6 (Eagle/Finch) architectures have several improvements based on the RWKV-4 architecture. Therefore, these two architectures are published in the same paper.

• RWKV 5/6 Paper: "Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence"
• Paper Link: arXiv:2404.05892

RWKV-4 is the first official version of the RWKV model. The paper was co-authored by Bo Peng and the RWKV community and was first published on May 22, 2023. In October of the same year, the RWKV-4 architecture paper was accepted by EMNLP 2023.

• RWKV-4 Paper: "RWKV: Reinventing RNNs for the Transformer Era"
• Paper Link: arXiv:2305.13048

RWKV has released open-source models of various parameter scales for each architecture version.

Here's the English translation:

::: tip Due to performance issues caused by outdated architectures, all RWKV-5, RWKV-4 series models (Raven / World / Pile ...) and earlier RWKV versions have reached end-of-life, with existing models serving only as archives. :::

• Models with 7B parameters and below: Please use the RWKV7-G1 2.9B model. Thanks to the latest architectural design and the newest World v3.5 dataset (totaling 5.16T tokens), the RWKV7-G1 model has strong reasoning, coding, and mathematical capabilities, performing better than previous models across all tasks.

• Models with more than 7B parameters: Since RWKV7-G1 7B and larger models are currently in training, we recommend using the RWKV-6-World-14B-V2.1 model.

::: tip After the RWKV7-G1 7B/14B models complete training, they will completely replace the existing RWKV-6-World 7B/14B models. :::

• Advantages

  • Lower resource usage during runtime and training (VRAM, CPU, GPU, etc.).
  • 10 to 100 times lower computational requirements compared to Transformers with larger contexts.
  • Supports linear scaling to any context length (Transformers scale quadratically).
  • Performs as well as Transformer architectures in terms of answer quality and generalization ability.
  • RWKV models' training data includes languages other than English (e.g., Chinese, Japanese, etc.), offering better multilingual capabilities than most existing open-source models.

• Disadvantages

  • RWKV base models are very sensitive to the format of prompts, and the format of prompts significantly affects the generation results.
  • Due to architectural design, RWKV models are weaker on tasks requiring lookback/review, and we are working on various optimizations to address this issue.

RWKV models typically have two naming conventions:

• RWKV-6-World-3B-v2.1-20240208-ctx4096.pth
• RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth

The meaning of each field in the model name:

RWKV is made possible, as an Open Source project, thanks to the large amount of GPU compute and researchers time contributions from

Without their invaluable support, we would not have been able to develop the core RWKV foundation models that you see today.

In addition, we would like to thank

• alpin @ pygmalionAI
• AutoMeta @ AlignmentLab
• FeatherlessAI
• Various other folks who donated slices of GPU time / preferred not to be named

For helping with GPU time, on smaller experiments, finetunes, and various models. Especially for those models that never get publically released in failed runs.