Description
1、OpenAI格式的接口无法在Cline中使用;包括VL和300B都无法正常工作;后台五明显报错日志;
2、后台日志过于简陋,无法观测性能指标;还是有什么特殊日志文件?
SGlang或VLLM都有显示token生成速度等功能,但是fastdeploy暂时没有找到;
3、并发10个情况下,很容易出现一些莫名其妙的回复:
如一直输出,查看后台有报错日志:
Traceback (most recent call last):
File "/usr/lib/python3.10/logging/init.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/init.py", line 943, in format
return fmt.format(record)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/utils.py", line 62, in format
message = super().format(record)
File "/usr/lib/python3.10/logging/init.py", line 678, in format
record.message = record.getMessage()
File "/usr/lib/python3.10/logging/init.py", line 368, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/output/token_processor.py", line 153, in process_sampling_results
self._process_batch_output()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/output/token_processor.py", line 282, in _process_batch_output
llm_logger.info(
Message: 'recovery stop signal found at task chatcmpl-4d7a5a03-02f4-4cb9-b63e-8d19e5394274-8a49a2d5-7dcc-44ea-96eb-113544957f4d'
Arguments: ('token_ids: [-3]',)