Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Jan 2, 2023

• [TP] Put parallel embeddings in separate modules

  tridao

  committed
  4cab4de

  Copy full SHA for 4cab4de

• [FusedDense] Limit matrix dims to 2M (instead of 64k)

  tridao

  committed
  1ec09eb

  Copy full SHA for 1ec09eb

Commits on Jan 1, 2023

• [Bert] Fix embedding layer norm before embedding dropout

  tridao

  committed
  714c1b4

  Copy full SHA for 714c1b4

• [GPT] Refactor function to shard state_dict for TensorParallel

  tridao

  committed
  ef1ba91

  Copy full SHA for ef1ba91

• [FusedDense] Kick off input all_gather before weight dtype conversion

  tridao

  committed
  65b4064

  Copy full SHA for 65b4064

• [Loss] Use flash_attn.losses.cross_entropy.CrossEntropyLoss

  tridao

  committed
  71befc1

  Copy full SHA for 71befc1

Commits on Dec 30, 2022

• [Docker] Set torchmetrics==0.10.3

  tridao

  committed
  cadfa39

  Copy full SHA for cadfa39

• [Docs] Fix formatting

  tridao

  committed
  4379896

  Copy full SHA for 4379896

• [Docs] Mention that dropout_layer_norm supports all dims up to 6k

  tridao

  committed
  3c7cbfc

  Copy full SHA for 3c7cbfc

• [Docs] Mention that XPos's scale_base is recommended to be 512

  tridao

  committed
  85b8e3d

  Copy full SHA for 85b8e3d

Commits on Dec 29, 2022

• Update training Dockerfile to use flash-attn==0.2.6

  tridao

  committed
  984d520

  Copy full SHA for 984d520

Commits on Dec 28, 2022

• Merge pull request #95 from Quentin-Anthony/patch-1

  Show description for 0296171
  tridao

  authored
  0296171

  Copy full SHA for 0296171

• Add gpt-neox adoption

  Quentin-Anthony

  authored
  d2a69a5

  Copy full SHA for d2a69a5

• Bump to v0.2.6

  tridao

  committed
  a6ec178

  Copy full SHA for a6ec178

• Implement generation for GPT

  tridao

  committed
  63670fd

  Copy full SHA for 63670fd

Commits on Dec 27, 2022

• Support loading GPT2 weights from Huggingface

  tridao

  committed
  9d797d8

  Copy full SHA for 9d797d8

• Tweak CrossEntropyLoss to take process_group in init

  tridao

  committed
  c6ecd40

  Copy full SHA for c6ecd40

• Implement Tensor Parallel for GPT model

  tridao

  committed
  b4018a5

  Copy full SHA for b4018a5

Commits on Dec 25, 2022

• Implement Tensor Parallel for GPT2Embeddings

  tridao

  committed
  78225c5

  Copy full SHA for 78225c5

• Implement Tensor Parallel for transformer Block

  tridao

  committed
  a8cfe51

  Copy full SHA for a8cfe51

• Implement TensorParallel for MHA

  tridao

  committed
  1e712ea

  Copy full SHA for 1e712ea

Commits on Dec 24, 2022

• Implement TensorParallel for FusedDense and FusedDenseGeluDense

  tridao

  committed
  226a1b7

  Copy full SHA for 226a1b7

Commits on Dec 23, 2022

• Add smoothing for CrossEntropyParallel, rename to CrossEntropyLoss

  tridao

  committed
  dff68c2

  Copy full SHA for dff68c2

• Simplify FusedDense

  tridao

  committed
  e68ebbe

  Copy full SHA for e68ebbe

Commits on Dec 21, 2022

• Bump to v0.2.5

  tridao

  committed
  1bc6e5b

  Copy full SHA for 1bc6e5b

• Implement XPos (Sun et al.)

  tridao

  committed
  496e4f5

  Copy full SHA for 496e4f5

• Fix typo in config: train.gpu -> train.gpu_mem

  tridao

  committed
  c2407de

  Copy full SHA for c2407de

Commits on Dec 20, 2022

• Implement last_layer_subset optimization for BERT

  tridao

  committed
  13cdceb

  Copy full SHA for 13cdceb

Commits on Dec 19, 2022

• Implement BERT

  tridao

  committed
  5fb6df0

  Copy full SHA for 5fb6df0

Commits on Dec 17, 2022

• Merge pull request #92 from ploshkin/rm-shape-asserts

  Show description for dc24c22
  tridao

  authored
  dc24c22

  Copy full SHA for dc24c22

• add asserts for sin shape

  ploshkin

  committed
  ee8984d

  Copy full SHA for ee8984d

Commits on Dec 16, 2022

• fix slicing dimensions

  ploshkin

  committed
  c7c6697

  Copy full SHA for c7c6697

• [Docs] Mention Megatron-LM

  tridao

  committed
  b78f5a3

  Copy full SHA for b78f5a3

• [Docs] Mention PubMedGPT

  tridao

  committed
  ece8f05

  Copy full SHA for ece8f05

Commits on Dec 15, 2022

• Remove redundant shape asserts in rotary embeddings

  ploshkin

  committed
  96656b9

  Copy full SHA for 96656b9

Pagination

Previous

Next