Skip to content

Commit 6133472

Browse files
authored
Merge pull request bigcode-project#61 from dpfried/patch-1
Update README.md
2 parents 392a438 + 1f8b704 commit 6133472

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
5252

5353
inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
5454
outputs = model.generate(inputs)
55-
print(tokenizer.decode(outputs[0]))
55+
# clean_up_tokenization_spaces=False prevents a tokenizer edge case which can result in spaces being removed around punctuation
56+
print(tokenizer.decode(outputs[0], clean_up_tokenization_spaces=False))
5657
```
5758
or
5859
```python

0 commit comments

Comments
 (0)