Chatting in CJK language lead to crash

[omegacoleman]

Case 1

POST /v1/chat/completions                                                                                                                                                  
🔹<|start_header_id|>system<|end_header_id|>

You are an expert story writer. Help me build a unique world for my fantasy novel. Describe the geography, cultures, political systems, and magical elements. Include details 
on key locations, major events in history, and significant societal norms. <|end_of_text|><|start_header_id|>user<|end_header_id|>

请使用中文，写至少5000字的小说，题材不限。请只输出中文，不要换行、输出列表，也不要输出任何拉丁字母。<|end_of_text|><|start_header_id|>assistant<|end_header_id|>

🔸我们的世界是这样子的。

第一章：地理

Assertion failed: strBufferPos + pieceLen + 1 < strBufferSize (src/tokenizer.cpp: decode: 225)
Aborted

Case 2

  POST /v1/chat/completions
🔹<|start_header_id|>system<|end_header_id|>

You are an expert story writer. Help me build a unique world for my fantasy novel. Describe the geography, cultures, political systems, and magical elements. Include details 
on key locations, major events in history, and significant societal norms. <|end_of_text|><|start_header_id|>user<|end_header_id|>

请使用中文，写至少5000字的小说，题材不限。请只输出中文并大量添加emoji，不要换行、输出列表，也不要输出任何拉丁字母。<|end_of_text|><|start_header_id|>assistant<|end_header_id|
>

🔸🌟 一个奇幻世界的诞生 🌟

🏞️ 地理

这个世界叫做艾尔达拉，艾尔达拉...多种多样，多种 力争夺权力。

🔮  Critical error: [json.exception.type_error.316] incomplete UTF-8 string; last byte: 0xE9

[omegacoleman]

After inspecting this issue for a few days, I proposed a fix in #228 .