Skip to content

Commit 63f0a2b

Browse files
Merge pull request stanfordnlp#1147 from fivejjs/main
updated vLLM local client usage in docs markdown
2 parents b2a9ae4 + acb1213 commit 63f0a2b

File tree

1 file changed

+10
-1
lines changed
  • docs/api/local_language_model_clients

1 file changed

+10
-1
lines changed

docs/api/local_language_model_clients/vLLM.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,16 @@ Initialize the `HFClientVLLM` within your program with the desired parameters. H
2525
```python
2626
lm = dspy.HFClientVLLM(model="mosaicml/mpt-7b", port=8000, url="http://localhost")
2727
```
28-
2928
Customize the `model`, `port`, `url`, and `max_tokens` according to your requirements. The `model` parameter should be set to the specific Hugging Face model ID you wish to use.
3029

3130
Please refer to the [official vLLM repository](https://github.com/vllm-project/vllm) for more detailed information and documentation.
31+
32+
### Sending requests to vLLM server using [dspy.OpenAI](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI)
33+
34+
Query the vLLM server using OpenAI SDK through [`dspy.OpenAI`](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI) with your desired parameters. Here is an example call:
35+
36+
```python
37+
lm = dspy.OpenAI(model="mosaicml/mpt-7b", api_base="http://localhost:8000/v1/", api_key="EMPTY")
38+
```
39+
40+
Similarly, customize the `model`, `port`, `url` (vLLM arguments), along with the remaining OpenAI client arguments according to your requirements. The `model` parameter should be set to the specific Hugging Face model ID you wish to use.

0 commit comments

Comments
 (0)