We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
2 parents 392a438 + 1f8b704 commit 6133472Copy full SHA for 6133472
README.md
@@ -52,7 +52,8 @@ model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
52
53
inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
54
outputs = model.generate(inputs)
55
-print(tokenizer.decode(outputs[0]))
+# clean_up_tokenization_spaces=False prevents a tokenizer edge case which can result in spaces being removed around punctuation
56
+print(tokenizer.decode(outputs[0], clean_up_tokenization_spaces=False))
57
```
58
or
59
```python
0 commit comments