You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,33 +10,33 @@ Create a new conda environment and activate it
10
10
conda create -n env
11
11
conda activate env
12
12
```
13
-
Find the pytorch version compatible with your version of cuda [here](https://pytorch.org/get-started/previous-versions/), for example the following command work with cuda 11.6
13
+
Install the `pytorch` version compatible with your version of cuda [here](https://pytorch.org/get-started/previous-versions/), for example the following command works with cuda 11.6
Install the latest version of peft by cloning the repository
32
+
Install the latest version of `peft` by cloning the repository
33
33
34
34
```bash
35
35
git clone https://github.com/huggingface/peft
36
36
cd peft
37
37
pip install -e .
38
38
```
39
-
Finally, install bitsandbytes and wandb
39
+
Finally, install `bitsandbytes` and `wandb`
40
40
```bash
41
41
pip install bitsandbytes
42
42
pip install wandb
@@ -49,10 +49,11 @@ Before you run any of the scripts make sure you are logged in and can push to th
49
49
```bash
50
50
huggingface-cli login
51
51
```
52
-
Make sure you are logged in wandb:
52
+
Make sure you are logged in `wandb`:
53
53
```bash
54
54
wandb login
55
55
```
56
+
Now that everything is done, you can clone the repository and get into the corresponding directory.
56
57
57
58
## Fine-Tuning (`finetune.py`)
58
59
💫 StarCoder can be fine-tuned to achieve multiple downstream tasks. Our interest here is to fine-tune StarCoder in order to make it follow instructions. [Instruction fine-tuning](https://arxiv.org/pdf/2109.01652.pdf) has gained a lot of attention recently as it proposes a simple framework that teaches language models to align their outputs with human needs. That procedure requires the availability of quality instruction datasets, which contain multiple `instruction - answer` pairs. Unfortunately such datasets are not ubiquitous but thanks to Hugging Face 🤗's [datasets](https://github.com/huggingface/datasets) library we can have access to some good proxies. To fine-tune cheaply and efficiently, we use Hugging Face 🤗's [PEFT](https://github.com/huggingface/peft) as well as Tim Dettmers' [bitsandbytes](https://github.com/TimDettmers/bitsandbytes).
0 commit comments