[CI][Spec Decode] Adjust threshold for flaky ngram spec decoding test again

[wwl2755]

tests/v1/e2e/test_spec_decode.py::test_ngram_correctness is still flaky after #24528

Example failures at main:
https://buildkite.com/vllm/ci/builds/30544#01993eac-288c-4699-af07-f991b95918c0,
https://buildkite.com/vllm/ci/builds/30529#01993e6c-3c5f-4de7-a7b2-cdd4ca680d6d,
https://buildkite.com/vllm/ci/builds/30507#01993dac-fe3a-4f99-b106-97c65b59a783,
https://buildkite.com/vllm/ci/builds/30434#01993aeb-c38f-4277-b736-f740bf4e548a

I ran it locally multiple times (>5) and the least number I have seen was 66.

Related: #24314

CC: @njhill @markmc @ekagra-ranjan @WoosukKwon @LiuXiaoxuanPKU

[gemini-code-assist]

Code Review

This pull request adjusts the threshold for a flaky test related to n-gram speculative decoding to fix CI failures. My review highlights a concern that this is a temporary fix for a potentially deeper issue. The test uses deterministic sampling parameters (temperature=0) but shows non-deterministic behavior, which could point to a bug. I've added a comment recommending an investigation into the root cause of the flakiness rather than repeatedly lowering the test's threshold, as this could hide future regressions.

[gemini-code-assist]

While lowering the threshold might fix the immediate flakiness in CI, this is the second time this has been done for this test. This pattern of repeatedly reducing the threshold is concerning as it can mask underlying bugs or performance regressions in the speculative decoding implementation.

Given that temperature=0 is used, the model's output should be deterministic. The fact that there are discrepancies between the reference and speculative outputs suggests a potential issue.

Instead of just lowering the threshold, it would be better to investigate the root cause of this non-determinism. Possible areas to investigate:

• Are there any non-deterministic operations in the model or CUDA kernels that are not properly seeded or configured for deterministic execution?
• Could there be floating-point precision differences that are causing the outputs to diverge?
• Is there a subtle bug in the n-gram speculative decoding logic that causes it to not be equivalent to the reference implementation in some cases?

Addressing the root cause would lead to a more robust and reliable test.

[njhill]

Thanks @wwl2755.

We should really fix the nondeterminism here though!

Perhaps we could look at running this particular test with fp32 (and flex attention if that works with spec decode?)

Or otherwise hopefully #24583 could help us!

[wwl2755]

We should really fix the nondeterminism here though!

Totally agreed with you!

Perhaps we could look at running this particular test with fp32 (and flex attention if that works with spec decode?)

Or otherwise hopefully #24583 could help us!

But actually I think ngram proposer would only run on CPU and should not generate any randomness? The biggest suspect I have right now is the rejection sampler part.

I can open up a issue to track it. (#24777)

[DarkLight1337]

Retrying flaky v1/entrypoints/openai/test_multi_api_servers.py::test_single_completion[ibm-research/PowerMoE-3b] - RuntimeError: Server failed to start in time

This test didn't fail that much before, I wonder what changed?

[DarkLight1337]

It passed this time

[wwl2755]

Retrying flaky v1/entrypoints/openai/test_multi_api_servers.py::test_single_completion[ibm-research/PowerMoE-3b] - RuntimeError: Server failed to start in time

This test didn't fail that much before, I wonder what changed?

Thanks for retrying that! I got no idea what exactly caused this. It seems something randomly went wrong during/after loading the weights. Anyway, I think it shouldn't be related to the change in this PR. 😂