Skip to content

Commit b177fc4

Browse files
alexrudallalexrudall
committed
Add Groq to README & specs
1 parent b060d86 commit b177fc4

File tree

4 files changed

+189
-9
lines changed

4 files changed

+189
-9
lines changed

‎README.md

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ Stream text with GPT-4, transcribe and translate audio with Whisper, or create i
2727
- [Faraday middleware](#faraday-middleware)
2828
- [Azure](#azure)
2929
- [Ollama](#ollama)
30+
- [Groq](#groq)
3031
- [Counting Tokens](#counting-tokens)
3132
- [Models](#models)
3233
- [Chat](#chat)
@@ -239,6 +240,27 @@ client.chat(
239240
# => Hi! It's nice to meet you. Is there something I can help you with, or would you like to chat?
240241
```
241242

243+
#### Groq
244+
245+
[Groq API Chat](https://console.groq.com/docs/quickstart) is broadly compatible with the OpenAI API, with a [few minor differences](https://console.groq.com/docs/openai). Get an access token from [here](https://console.groq.com/keys), then:
246+
247+
````ruby
248+
client = OpenAI::Client.new(
249+
access_token: "groq_access_token_goes_here",
250+
uri_base: "https://api.groq.com/"
251+
)
252+
253+
client.chat(
254+
parameters: {
255+
model: "llama3", # Required.
256+
messages: [{ role: "user", content: "Hello!"}], # Required.
257+
temperature: 0.7,
258+
stream: proc do |chunk, _bytesize|
259+
print chunk.dig("choices", 0, "delta", "content")
260+
end
261+
})
262+
```
263+
242264
### Counting Tokens
243265

244266
OpenAI parses prompt text into [tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them), which are words or portions of words. (These tokens are unrelated to your API access_token.) Counting tokens can help you estimate your [costs](https://openai.com/pricing). It can also help you ensure your prompt text size is within the max-token limits of your model's context window, and choose an appropriate [`max_tokens`](https://platform.openai.com/docs/api-reference/chat/create#chat/create-max_tokens) completion parameter so your response will fit as well.
@@ -247,7 +269,7 @@ To estimate the token-count of your text:
247269
248270
```ruby
249271
OpenAI.rough_token_count("Your text")
250-
```
272+
````
251273
252274
If you need a more accurate count, try [tiktoken_ruby](https://github.com/IAPark/tiktoken_ruby).
253275

‎spec/fixtures/cassettes/groq_llama3-8b-8192_streamed_chat.yml

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
http_interactions:
3+
- request:
4+
method: post
5+
uri: https://api.groq.com/openai/v1/chat/completions
6+
body:
7+
encoding: UTF-8
8+
string: '{"model":"llama3-8b-8192","messages":[{"role":"user","content":"Hello!"}],"stream":true}'
9+
headers:
10+
Content-Type:
11+
- application/json
12+
Authorization:
13+
- Bearer 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
14+
Accept-Encoding:
15+
- gzip;q=1.0,deflate;q=0.6,identity;q=0.3
16+
Accept:
17+
- "*/*"
18+
User-Agent:
19+
- Ruby
20+
response:
21+
status:
22+
code: 200
23+
message: OK
24+
headers:
25+
Date:
26+
- Sun, 28 Apr 2024 12:58:48 GMT
27+
Content-Type:
28+
- text/event-stream
29+
Transfer-Encoding:
30+
- chunked
31+
Connection:
32+
- keep-alive
33+
Cache-Control:
34+
- no-cache
35+
Vary:
36+
- Origin
37+
X-Ratelimit-Limit-Requests:
38+
- '14400'
39+
X-Ratelimit-Limit-Tokens:
40+
- '12000'
41+
X-Ratelimit-Remaining-Requests:
42+
- '14399'
43+
X-Ratelimit-Remaining-Tokens:
44+
- '11994'
45+
X-Ratelimit-Reset-Requests:
46+
- 6s
47+
X-Ratelimit-Reset-Tokens:
48+
- 30ms
49+
X-Request-Id:
50+
- req_01hwjcjqzaebarxz5f2vm7fz34
51+
Via:
52+
- 1.1 google
53+
Alt-Svc:
54+
- h3=":443"; ma=86400
55+
Cf-Cache-Status:
56+
- DYNAMIC
57+
Set-Cookie:
58+
- __cf_bm=.ZNFRsf2P9DwyqjhO3cuf9Pu_3fVQzLXYXtUcJ_UMkI-1714309128-1.0.1.1-w6OhIRMxkM4Ouz9NIqUiuv8Kd1o9nuCib2x07qAHI12yNx193y6_5ol27ayr.mfxkd0ME.JLgC3W5DGsXoA3SA;
59+
path=/; expires=Sun, 28-Apr-24 13:28:48 GMT; domain=.groq.com; HttpOnly; Secure;
60+
SameSite=None
61+
Server:
62+
- cloudflare
63+
Cf-Ray:
64+
- 87b739d22ceb1c23-SOF
65+
body:
66+
encoding: UTF-8
67+
string: |+
68+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"x_groq":{"id":"req_01hwjcjqzaebarxz5f2vm7fz34"}}
69+
70+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}]}
71+
72+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}]}
73+
74+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" It"},"logprobs":null,"finish_reason":null}]}
75+
76+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":"'s"},"logprobs":null,"finish_reason":null}]}
77+
78+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" nice"},"logprobs":null,"finish_reason":null}]}
79+
80+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" to"},"logprobs":null,"finish_reason":null}]}
81+
82+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" meet"},"logprobs":null,"finish_reason":null}]}
83+
84+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}
85+
86+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":"."},"logprobs":null,"finish_reason":null}]}
87+
88+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" Is"},"logprobs":null,"finish_reason":null}]}
89+
90+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" there"},"logprobs":null,"finish_reason":null}]}
91+
92+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" something"},"logprobs":null,"finish_reason":null}]}
93+
94+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" I"},"logprobs":null,"finish_reason":null}]}
95+
96+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" can"},"logprobs":null,"finish_reason":null}]}
97+
98+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" help"},"logprobs":null,"finish_reason":null}]}
99+
100+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}
101+
102+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" with"},"logprobs":null,"finish_reason":null}]}
103+
104+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":","},"logprobs":null,"finish_reason":null}]}
105+
106+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" or"},"logprobs":null,"finish_reason":null}]}
107+
108+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" would"},"logprobs":null,"finish_reason":null}]}
109+
110+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}
111+
112+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" like"},"logprobs":null,"finish_reason":null}]}
113+
114+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" to"},"logprobs":null,"finish_reason":null}]}
115+
116+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":" chat"},"logprobs":null,"finish_reason":null}]}
117+
118+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{"content":"?"},"logprobs":null,"finish_reason":null}]}
119+
120+
data: {"id":"chatcmpl-020c62b8-ef08-4c4f-8d11-5fe27c0ad788","object":"chat.completion.chunk","created":1714309128,"model":"llama3-8b-8192","system_fingerprint":"fp_dadc9d6142","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"x_groq":{"id":"req_01hwjcjqzaebarxz5f2vm7fz34","usage":{"queue_time":0.085418399,"prompt_tokens":12,"prompt_time":0.005,"completion_tokens":25,"completion_time":0.029,"total_tokens":37,"total_time":0.034}}}
121+
122+
data: [DONE]
123+
124+
recorded_at: Sun, 28 Apr 2024 12:58:48 GMT
125+
recorded_with: VCR 6.1.0
126+
...

‎spec/fixtures/cassettes/llama3_chat.yml renamed to ‎spec/fixtures/cassettes/ollama_llama3_chat.yml

Lines changed: 7 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎spec/openai/client/chat_spec.rb

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@
1111
end
1212
let(:parameters) { { model: model, messages: messages, stream: stream } }
1313
let(:content) { response.dig("choices", 0, "message", "content") }
14-
let(:cassette) { "#{model} #{'streamed' if stream} chat".downcase }
14+
let(:provider) { nil }
15+
let(:cassette) { "#{"#{provider}_" if provider}#{model} #{'streamed' if stream} chat".downcase }
1516

1617
context "with model: gpt-3.5-turbo" do
1718
let(:model) { "gpt-3.5-turbo" }
@@ -176,6 +177,7 @@ def call(chunk)
176177

177178
context "with Ollama + model: llama3" do
178179
let(:uri_base) { "http://localhost:11434" }
180+
let(:provider) { "ollama" }
179181
let(:model) { "llama3" }
180182

181183
it "succeeds" do
@@ -190,6 +192,36 @@ def call(chunk)
190192
end
191193
end
192194
end
195+
196+
context "with Groq + model: llama3" do
197+
let(:uri_base) { "https://api.groq.com/openai" }
198+
let(:provider) { "groq" }
199+
let(:model) { "llama3-8b-8192" }
200+
let(:access_token) { ENV.fetch("GROQ_ACCESS_TOKEN", nil) }
201+
let(:response) do
202+
OpenAI::Client.new({ uri_base: uri_base, access_token: access_token }).chat(
203+
parameters: parameters
204+
)
205+
end
206+
let(:chunks) { [] }
207+
let(:stream) do
208+
proc do |chunk, _bytesize|
209+
print chunk["choices"][0]["delta"]["content"]
210+
chunks << chunk
211+
end
212+
end
213+
214+
it "succeeds" do
215+
VCR.use_cassette(cassette) do
216+
unless access_token
217+
pending "This test needs a `GROQ_ACCESS_TOKEN` environment variable to run"
218+
end
219+
220+
response
221+
expect(chunks.dig(0, "choices", 0, "index")).to eq(0)
222+
end
223+
end
224+
end
193225
end
194226
end
195227
end

0 commit comments

Comments
 (0)