-
-
Notifications
You must be signed in to change notification settings - Fork 10.6k
[Frontend] enable custom logging for the uvicorn server (OpenAI API server) #18403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
Hello @russellb, let me know if you need additional context on this PR or if you have remarks to address to make it green for merge |
Let me know @DarkLight1337 if I can add anything to ease the review to make sure this does not diverge from |
hi @DarkLight1337 - I re-ran the CI and resolved the conflict with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to a separate function so we can inline the log_config
argument into the existing code at L1336
12a0fc6
to
8f86dd9
Compare
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
Signed-off-by: François Paupier <[email protected]>
ensure it works with no log file provided vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
Signed-off-by: François Paupier <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: François Paupier <[email protected]>
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
ssl_cert_reqs=args.ssl_cert_reqs, | ||
**uvicorn_kwargs, | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary change
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed 👌
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
vllm-project#18210 Signed-off-by: François Paupier <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, sorry for the delay!
…erver) (vllm-project#18403) Signed-off-by: François Paupier <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>
This PR introduce the possibility to define a custom logging configuration for the uvicorn server used to expose the OpenAI API compliant API.
Previously it was possible to modify the log format of the vLLM engine but specifying a config file had no effect over the uvicorn API.
Hence you always saw something like this uvicorn log format when starting your app (log below)
This is limitating for use in production systems that have auditing requirement and must expose logs in a specific format to ensure compliance or just to make it easier to parse logs from various systems.
This PR extend the custom logging possibility defined in https://docs.vllm.ai/en/latest/getting_started/examples/logging_configuration.html to enable custom uvicorn logging format.
FIX #18210
You can test it yourself with the provided logger config below and making sure to expose it to the vLLM process by doing
export VLLM_LOGGING_CONFIG_PATH=./logger-config.json vllm serve....
You will see the following log at service startup for the uvicorn server: