Issue reproducing Float8 performance benchmark

[xmfan]

Bug description

I'm looking at https://github.com/pytorch/torchtitan/blob/main/benchmarks/llama3_h100_202412_torchtitan.md. Specifically, this table:

I'm not certain what the repro command for this. From https://github.com/pytorch/torchtitan/blob/main/docs/float8.md, I went ahead with CONFIG_FILE="./torchtitan/models/llama3/train_configs/llama3_8b.toml" ./run_train.sh --model.converters="float8" --float8.enable_fsdp_float8_all_gather --float8.precompute_float8_dynamic_scale_for_fsdp --float8.force_recompute_fp8_weight_in_bwd --training.compile.

Made the following changes to my llama3 toml: https://gist.github.com/xmfan/53fca4ed56cf7e713a282ce6e1922e9e

• seq_len = 32768
• data_parallel_shard_degree = 8 (for 8 gpu fsdp)
• activation_checkpoint.mode = "full"
• steps = 400 (just for a shorter run)

But my peak memory of the run seems way lower than the one quoted in the perf benchmarks, which makes me think I did something wrong. @tianyu-l tried these settings, and got a hang instead.

Are these the correct settings for this benchmark?

https://gist.github.com/xmfan/5a6b6daa0968aed7499ef364dae61420

Versions

latest torchao (USE_CPP=0 python -m pip install git+https://github.com/pytorch/ao.git), pytorch 06/25 nightly, torchtitan main

[tianyu-l]

cc @danielvegamyhre

[danielvegamyhre]

Hmm, I didn't produce this benchmark data but I can take a look at this.

[tianyu-l]

@danielvegamyhre The benchmark was done by me. I tried to reproduce but I got hang for llama3-8b.
@xmfan got less than half memory than the numbers listed in the table, so I think it's pretty surprising

[danielvegamyhre]

I think the original benchmark had a bug or perhaps was run on a shared devgpu where someone else was using GPU memory as well, because i have never seen nearly maxing out global memory (84 / 95gb) when parallelizing across 8 GPUs w/ FSDP, AND using Full AC, AND using compile. (all 3 of those lower peak memory). That would indicate a bug or another process using GPU memory.

[danielvegamyhre]

I ran this and got 45.3GB peak memory, 4,289 TPS. cc @tianyu-l @xmfan

• llama3 8b toml copied from @xmfan's here: https://gist.github.com/xmfan/53fca4ed56cf7e713a282ce6e1922e9e
• command copied from issue description: CONFIG_FILE="./torchtitan/models/llama3/train_configs/llama3_8b.toml" ./run_train.sh --model.converters="float8" --float8.enable_fsdp_float8_all_gather --float8.precompute_float8_dynamic_scale_for_fsdp --float8.force_recompute_fp8_weight_in_bwd --training.compile

Unless i'm missing something, this looks like a classic symptom of running benchmarks while GPUs are being used by other processes as well (i.e., higher peak memory, lower TPS). If others agree, we should update the benchmarks and then we can close the issue.

[tianyu-l]

@danielvegamyhre

Thanks for looking into this. I also tried and obtained similar results to yours.

I can't remember but it could be due to I ran them on a shared devgpu.

Could you help update Table 1 and Table 2?
The point of Table 1 is to showcase max throughput on 8 GPUs with / without torch.compile and float8
The point of Table 2 is to showcase longest seq len on 8 GPUs with / without CP -- so we can multiple 2 (or 4 or whatever could fit in memory) for every entry in the table

Also we'd need to have a note stating that this is redone recently 2025/07.