Update llama4 documentation to use the right settings for long context (#48)

Author: luccafong

_posts/2025-04-05-llama4.md

@@ -25,35 +25,39 @@ On 8x H100 GPUs:
 * Scout (up to 1M context):
 
 ```
-vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
+VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
   --tensor-parallel-size 8 \
-  --max-model-len 1000000
+  --max-model-len 1000000 --override-generation-config='{"attn_temperature_tuning": true}'
 ```
 
 * Maverick (up to \~430K context):
 
 ```
-vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
+VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
   --tensor-parallel-size 8 \
-  --max-model-len 430000
+  --max-model-len 430000 --override-generation-config='{"attn_temperature_tuning": true}'
 ```
 
 On 8x H200 GPUs:
 
 * Scout (up to 3.6M context):
 
 ```
-vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
+VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
   --tensor-parallel-size 8 \
-  --max-model-len 3600000
+  --max-model-len 3600000 --override-generation-config='{"attn_temperature_tuning": true}'
 ```
 
 * Maverick (up to 1M context):
 
 ```
-vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
+VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
   --tensor-parallel-size 8
+  --max-model-len 1000000 --override-generation-config='{"attn_temperature_tuning": true}'
 ```
+
+Note: we highly recommend to turn on attn_temperature_tuning to improve accuracy for long contexts longer than 32K tokens, and VLLM_DISABLE_COMPILE_CACHE=1 is required.
+
 **Multimodality:**
 
 The Llama 4 models excel at image understanding up to 8-10 images. By default, vLLM server accepts 1 image per request. Please pass `--limit-mm-per-prompt image=10` to serve up to 10 images per request with OpenAI-compatible API. We also recommend checking out our multi-image offline inference example with Llama-4 [here](https://github.com/vllm-project/vllm/blob/v0.8.3/examples/offline_inference/vision_language_multi_image.py).
@@ -70,7 +74,6 @@ While more performance enhancements are on the way, we believe the Llama 4 model
 
 * **Boost Performance & Context Length:** Set `--kv-cache-dtype fp8` to potentially double the usable context window and gain a performance boost. We observe little to no accuracy drop in relevant evaluations with this setting.
 * **Maximize Context Window (up to 10M):** To fully utilize the maximum context windows (up to 10M for Scout), we recommend serving across multiple nodes using tensor parallelism or pipeline parallelism. Follow our distributed inference guide [here](https://docs.vllm.ai/en/latest/serving/distributed_serving.html).
-* **Improve Long Context Accuracy (\>32K):** We highly recommend adding `--override-generation-config='{"attn_temperature_tuning": true}'` to improve accuracy for contexts longer than 32K tokens.
 
 **Other Hardware Support & Quantizations:**