Phi3Tokenizer Chinese garbled code

[williamlzw]

Microsoft.ML.GenAI.Phi 0.23.0-preview.1.25127.4
Microsoft.ML.Tokenizers 2.0.0-preview.1.25127.4
Microsoft.ML.Tokenizers.Data.Cl100kBase 2.0.0-preview.1.25127.4

public async static void Test2()
{
    string device = "cuda";
    var weightFolder = @"D:\model\Phi-3-mini-128k-instruct";
    var model = Phi3ForCasualLM.FromPretrained(weightFolder, "config.json", layersOnTargetDevice: -1, quantizeToInt4: true, targetDevice: device);

    var modelPath = Path.Join(weightFolder, "tokenizer.model");
    var tokenizer = Phi3TokenizerHelper.FromPretrained(modelPath);

    var pipeline = new CausalLMPipeline<Tokenizer, Phi3ForCasualLM>(tokenizer, model, device);
    var client = new Phi3CausalLMChatClient(pipeline);
    var task = """
        你能讲一个有趣的笑话吗?
        """;
   
    List<ChatMessage> _chatHistory = new();
    _chatHistory.Add(new ChatMessage(ChatRole.System, "你是一个助手,用中文回答用户的问题"));
    _chatHistory.Add(new ChatMessage(ChatRole.User, task));
    var options = new ChatOptions
    {
        StopSequences = ["<|end_of_text|>"],//phi3
        AdditionalProperties = new() { { "max_length", 2048 } },
    };

    await foreach (var response in client.GetStreamingResponseAsync(_chatHistory, options))
    {
        Console.Write(response.Text);
    }

    Console.WriteLine();
    Console.WriteLine("End!");
    
}

output:
当然可以。有一位老???问一个小???：“???子们

[michaelgsharp]

@LittleLittleCloud can you take a look into this please?

[LittleLittleCloud]

@williamlzw Which console are you using,, could it be a cmd decoding issue?

[williamlzw]

Some Chinese characters are garbled, it should not be a console issue, it may be due to dictionary decoding.

[williamlzw]

#7441