Skip to content
Open
Changes from 1 commit
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
9b59d2e
[Automated Commit] Format Codebase
mlcommons-bot Dec 20, 2024
fe51c12
Merge branch 'mlcommons:master' into master
v-shobhit Sep 18, 2025
1f2666c
[Automated Commit] Format Codebase
github-actions[bot] Sep 18, 2025
f9c4f61
initial
v-shobhit Sep 27, 2025
db9d25e
[Automated Commit] Format Codebase
github-actions[bot] Sep 27, 2025
9daa72c
json fixes
v-shobhit Sep 27, 2025
2d0a179
updates, tokenizer
v-shobhit Sep 27, 2025
c8e679d
fix padding
v-shobhit Sep 27, 2025
50453c6
concurrent requests
v-shobhit Sep 27, 2025
e76b68d
increase timeout
v-shobhit Sep 27, 2025
b6d5671
[Automated Commit] Format Codebase
github-actions[bot] Sep 27, 2025
4fd4f56
add refactor changes
v-shobhit Sep 27, 2025
1df0885
[Automated Commit] Format Codebase
github-actions[bot] Sep 27, 2025
eb2f48c
rm truncation
v-shobhit Sep 27, 2025
4f35b8a
rm truncation, wait for server ready
v-shobhit Sep 27, 2025
354cb62
left padding
v-shobhit Sep 27, 2025
75f4307
[Automated Commit] Format Codebase
github-actions[bot] Sep 27, 2025
065bf7c
fixes
v-shobhit Sep 27, 2025
36aa581
add failure check
v-shobhit Sep 27, 2025
c75e629
change opts
v-shobhit Sep 27, 2025
040e986
organize files
v-shobhit Oct 1, 2025
991ff7e
rm submodule
v-shobhit Oct 1, 2025
492847a
add infer stuff
v-shobhit Oct 1, 2025
f3a3282
add harmonize-tokens.py
v-shobhit Oct 1, 2025
8596b13
move things
v-shobhit Oct 1, 2025
1db2f96
[Automated Commit] Format Codebase
github-actions[bot] Oct 1, 2025
8c08778
add README
v-shobhit Oct 1, 2025
168f210
fix name
v-shobhit Oct 1, 2025
13775b1
add commands
v-shobhit Oct 1, 2025
37c646d
update README
v-shobhit Oct 1, 2025
9cd0c61
accepts output_ids and detokenize
v-shobhit Oct 1, 2025
6abf8fa
[Automated Commit] Format Codebase
github-actions[bot] Oct 1, 2025
8fe8712
add plotter
v-shobhit Oct 1, 2025
889db8c
fix sampling
v-shobhit Oct 1, 2025
381cf60
add reasoning effort option
v-shobhit Oct 1, 2025
209e923
[Automated Commit] Format Codebase
github-actions[bot] Oct 1, 2025
fdc518d
doc option
v-shobhit Oct 1, 2025
f73da58
draw input histograms
v-shobhit Oct 1, 2025
4219a77
[Automated Commit] Format Codebase
github-actions[bot] Oct 1, 2025
782f066
updates
v-shobhit Oct 8, 2025
326a5fa
[Automated Commit] Format Codebase
github-actions[bot] Oct 8, 2025
1b99263
add more opts
v-shobhit Oct 8, 2025
579ef1c
move
v-shobhit Oct 13, 2025
ec8d12e
[Automated Commit] Format Codebase
github-actions[bot] Oct 13, 2025
a9de6f4
refactor
v-shobhit Oct 13, 2025
387779c
updates
v-shobhit Oct 13, 2025
bde219f
rename opt
v-shobhit Oct 13, 2025
24ef1e0
updates
v-shobhit Oct 14, 2025
141dd8d
[Automated Commit] Format Codebase
github-actions[bot] Oct 14, 2025
684e27a
add healthbench prompt creation
v-shobhit Oct 15, 2025
f5b04db
[Automated Commit] Format Codebase
github-actions[bot] Oct 15, 2025
d44256e
add healthbench eval
v-shobhit Oct 16, 2025
ff9133b
[Automated Commit] Format Codebase
github-actions[bot] Oct 16, 2025
5f1fd8e
add scripts to fetch datasets
v-shobhit Oct 16, 2025
ca654c8
[Automated Commit] Format Codebase
github-actions[bot] Oct 16, 2025
8c59f03
update requirements
v-shobhit Oct 28, 2025
108ea9d
add setup enroot script
v-shobhit Oct 28, 2025
f302c7c
add changes
v-shobhit Oct 29, 2025
534390c
[Automated Commit] Format Codebase
github-actions[bot] Oct 29, 2025
78bf971
add symlinks to gitmodules
v-shobhit Oct 29, 2025
115f498
add fetch_lcb.py
v-shobhit Oct 29, 2025
9571c11
[Automated Commit] Format Codebase
github-actions[bot] Oct 29, 2025
f8244ee
updates
v-shobhit Nov 4, 2025
531c37c
add pass@k; add spec decode option
v-shobhit Nov 5, 2025
5e86d65
add openai client; add pass@k
v-shobhit Nov 5, 2025
8c02839
[Automated Commit] Format Codebase
github-actions[bot] Nov 5, 2025
be4109e
restrict lcb to v5
v-shobhit Nov 5, 2025
c0b9ef3
[Automated Commit] Format Codebase
github-actions[bot] Nov 5, 2025
dc54e98
lcb optimizations
v-shobhit Nov 5, 2025
41cac5a
remove openai client
v-shobhit Nov 5, 2025
06a5387
[Automated Commit] Format Codebase
github-actions[bot] Nov 5, 2025
d927fb3
rename
v-shobhit Nov 10, 2025
c2fda5e
remove mmlu, healthbench
v-shobhit Nov 10, 2025
c5c389b
add fetch_all.py
v-shobhit Nov 10, 2025
341c750
updates
v-shobhit Nov 10, 2025
ef142a6
add preprocess
v-shobhit Nov 10, 2025
834175f
update README
v-shobhit Nov 10, 2025
e447081
[Automated Commit] Format Codebase
github-actions[bot] Nov 10, 2025
a1e668a
add top-p option
v-shobhit Nov 10, 2025
3db5bc2
add summarize_eval
v-shobhit Nov 10, 2025
3ca2cc1
[Automated Commit] Format Codebase
github-actions[bot] Nov 10, 2025
09cd3f7
add trtllm infer script
v-shobhit Nov 7, 2025
1aad396
fixes
v-shobhit Nov 7, 2025
a6e05f2
add round-robin for multi-dp
v-shobhit Nov 8, 2025
a2936a6
fix timeout issues
v-shobhit Nov 8, 2025
188411c
[Automated Commit] Format Codebase
github-actions[bot] Nov 8, 2025
b324d7d
add anthropic
v-shobhit Nov 11, 2025
f8a9f43
add stuff
v-shobhit Nov 11, 2025
8429244
optimize lcb multi-pass
v-shobhit Nov 11, 2025
d1f2794
[Automated Commit] Format Codebase
github-actions[bot] Nov 11, 2025
ee33969
rm healthbench
v-shobhit Nov 11, 2025
20f8916
[Automated Commit] Format Codebase
github-actions[bot] Nov 11, 2025
1e67e0c
lcb bug fixes
v-shobhit Nov 11, 2025
96c90e1
[Automated Commit] Format Codebase
github-actions[bot] Nov 11, 2025
e6d9c67
omit top-k if 0
v-shobhit Nov 11, 2025
ff892bb
[Automated Commit] Format Codebase
github-actions[bot] Nov 11, 2025
e4043d2
add changes and plotting scripts
v-shobhit Nov 12, 2025
1714c09
[Automated Commit] Format Codebase
github-actions[bot] Nov 12, 2025
19fcc80
add overall number
v-shobhit Nov 12, 2025
6cb7698
[Automated Commit] Format Codebase
github-actions[bot] Nov 12, 2025
b281727
add glob matching
v-shobhit Nov 12, 2025
9a0c45a
rename
v-shobhit Nov 20, 2025
35ea0e4
add pubmed tokenization
v-shobhit Nov 20, 2025
1c73423
updates
v-shobhit Nov 20, 2025
9a9194f
add tentative gpt-oss fields
v-shobhit Nov 20, 2025
1812c04
remove data dir
v-shobhit Nov 20, 2025
16af1bf
create preprocess module
v-shobhit Nov 20, 2025
9b4a84c
move things to archive
v-shobhit Nov 20, 2025
9450d46
[Automated Commit] Format Codebase
github-actions[bot] Nov 20, 2025
319e5f7
rm unused scripts
v-shobhit Nov 20, 2025
4d89b98
rm unused
v-shobhit Nov 20, 2025
be02519
mv things
v-shobhit Nov 20, 2025
18a8444
add mlperf artifacts
v-shobhit Nov 20, 2025
199f476
add mlperf artifacts
v-shobhit Nov 20, 2025
425ce75
[Automated Commit] Format Codebase
github-actions[bot] Nov 20, 2025
7cdc7cb
add utils, gitignore
v-shobhit Nov 20, 2025
ab90695
[Automated Commit] Format Codebase
github-actions[bot] Nov 20, 2025
62e4d47
update README
v-shobhit Nov 20, 2025
292c49d
fix request pool size
v-shobhit Nov 20, 2025
98585b8
[Automated Commit] Format Codebase
github-actions[bot] Nov 20, 2025
21a8034
add setup
v-shobhit Nov 21, 2025
734d8f4
updates
v-shobhit Nov 21, 2025
e3e22b8
server scenario fix; gpt-oss -> gpt-oss-120b
v-shobhit Nov 21, 2025
e40a7da
add fixes
v-shobhit Nov 21, 2025
382fc9e
add accuracy eval script for mlperf
v-shobhit Nov 21, 2025
31f435a
finishing touches
v-shobhit Nov 21, 2025
63592a3
[Automated Commit] Format Codebase
github-actions[bot] Nov 21, 2025
a41f882
refactor mode -> scenario
v-shobhit Nov 21, 2025
b2bc9e0
Merge branch 'mlcommons:master' into gptoss-loadgen
v-shobhit Nov 24, 2025
81f6ca5
add eval_perf script
v-shobhit Nov 26, 2025
f780189
[Automated Commit] Format Codebase
github-actions[bot] Nov 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add commands
  • Loading branch information
v-shobhit committed Oct 1, 2025
commit 13775b11ab6fdcdaffa2e6a2a46d08c77cf8a47e
15 changes: 14 additions & 1 deletion language/gpt-oss/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,12 @@ Some queries may have multi-shot examples. For these, the `User` and `Assistant`
SGLang docker image will be used: `lmsysorg/sglang:v0.5.3rc1`. Steps below are to be run in an environment from this image

### Preprocess the dataset

```bash
python3 harmonize_inputs.py \
--data-file mlperf_dsr1_fp8_ref_eval.pkl \
--num-processes 32 \
--output-file out/mlperf_gptoss_inputs.pkl
```

### Run the server
```bash
Expand All @@ -87,3 +92,11 @@ python3 -m sglang.launch_server \
--stream-interval 50
```

### Run the inference
```bash
python3 run_infer.py \
--input-tokens out/mlperf_gptoss_inputs.pkl \
--max-tokens 20480 \
--max-concurrency 4096 \
--output out/mlperf_gptoss_inferred.pkl
```