Issue with model weights when using api endpoints to run gradio_web_server_multi

[mb208]

Hi,

I'm trying to run the Chatbot Arena locally using my own API keys via the --register-api-endpoint-file flag. However, when I input any prompt in anonymous battle mode. Running the following script,

python3 -m fastchat.serve.gradio_web_server_multi \
    --controller "" \
    --register-api-endpoint-file api_endpoints.json \
    --share

gives me the following error:

ValueError: probabilities contain NaN

This seems to come from the following block in fastchat/serve/gradio_block_arena_anony.py, around line 228:

model_weights = []
    for model in models:
        weight = get_sample_weight(
            model, outage_models, sampling_weights, sampling_boost_models
        )
        model_weights.append(weight)
    total_weight = np.sum(model_weights)
    model_weights = model_weights / total_weight
    # print(models)
    # print(model_weights)
    chosen_idx = np.random.choice(len(models), p=model_weights)
    chosen_model = models[chosen_idx]

The problem is that model_weights ends up being empty or all zeros, so total_weight becomes 0, which leads to division by zero and NaN values.

From what I can tell outage_models, sampling_weights, and sampling_boost_models are empty or not initialized and I don't see a documented way to set weights through the api_endpoints.json file and have them properly integrated into this sampling logic

Is this expected? Should api_endpoints.json support something like model_weight, and if so, how should it be connected to this logic? Or is manual patching the only current workaround?

Thanks!

[Hoilap]

I met the same question, do you have the solutions? Thx! :)