@@ -13,50 +13,27 @@ Our investigations reveal that, through the cultivation and utilization of sub-m
1313
1414![ Performance on APPS] ( ./imgs/impression.png )
1515
16+
17+
1618** Performance on APPS**
17- | Model | Size | Pass@ | Introductory | Interview | Competition | All |
18- | ---------------------------| ------| -------| --------------| -----------| -------------| ------|
19- | ** GPT-Neo** | 2.7B | 1 | 3.90 | 0.57 | 0.00 | 1.12 |
20- | | | 5 | 5.50 | 0.80 | 0.00 | 1.58 |
21- | ** Codex** | 12B | 1 | 4.14 | 0.14 | 0.02 | 0.92 |
22- | | | 5 | 9.65 | 0.51 | 0.09 | 2.25 |
23- | | | 1000 | 25.02 | 3.70 | 3.23 | 7.87 |
24- | ** AlphaCode** | 1B | 1000 | 17.67 | 5.24 | 7.06 | 8.09 |
25- | ** AlphaCode (Filtered 1k)** | | 5 | 14.36 | 5.63 | 4.58 | 7.17 |
26- | ** AlphaCode (Filtered 10k)** | | 5 | 18.18 | 8.21 | 6.65 | 9.89 |
27- | ** AlphaCode (Filtered 50k)** | | 5 | 20.36 | 9.66 | 7.75 | 11.42 |
28- | ** StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
29- | ** WizardCoder** | 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
30- | ** CodeLlama** | 7B | 5 | 10.76 | 2.01 | 0.77 | 3.51 |
31- | | | 10 | 15.59 | 3.12 | 1.41 | 5.27 |
32- | | | 100 | 33.52 | 9.40 | 7.13 | 13.77|
33- | | 13B | 5 | 23.74 | 5.63 | 2.05 | 8.54 |
34- | | | 10 | 30.19 | 8.12 | 3.35 | 11.58|
35- | | | 100 | 48.99 | 18.40 | 11.98 | 23.23|
36- | | 34B | 5 | 32.81 | 8.75 | 2.88 | 12.39|
37- | | | 10 | 38.97 | 12.16 | 4.69 | 16.03|
38- | | | 100 | 56.32 | 24.31 | 15.39 | 28.93|
39- | ** CodeLlama-Python** | 7B | 5 | 12.72 | 4.18 | 1.31 | 5.31 |
40- | | | 10 | 18.50 | 6.25 | 2.24 | 7.90 |
41- | | | 100 | 38.26 | 14.94 | 9.12 | 18.44|
42- | | 13B | 5 | 26.33 | 7.06 | 2.79 | 10.06|
43- | | | 10 | 32.77 | 10.03 | 4.33 | 13.44|
44- | | | 100 | 51.60 | 21.46 | 14.60 | 26.12 |
45- | | 34B | 5 | 28.94 | 7.80 | 3.45 | 11.16 |
46- | | | 10 | 35.91 | 11.12 | 5.53 | 14.96 |
47- | | | 100 | 54.92 | 23.90 | 16.81 | 28.69 |
48- | ** CodeLlama-Instruct** | 7B | 5 | 12.85 | 2.07 | 1.13 | 4.04 |
49- | | | 10 | 17.86 | 3.12 | 1.95 | 5.83 |
50- | | | 100 | 35.37 | 9.44 | 8.45 | 14.43 |
51- | | 13B | 5 | 24.01 | 6.93 | 2.39 | 9.44 |
52- | | | 10 | 30.27 | 9.58 | 3.83 | 12.57 |
53- | | | 100 | 48.73 | 19.55 | 13.12 | 24.10 |
54- | | 34B | 5 | 31.56 | 7.86 | 3.21 | 11.67 |
55- | | | 10 | 37.80 | 11.08 | 5.12 | 15.23 |
56- | | | 100 | 55.72 | 22.80 | 16.38 | 28.10 |
19+
20+ | Model | Size | Pass@ | Introductory | Interview | Competition | All |
21+ | ------------| -------| -------| --------------| -----------| -------------| -------|
22+ | ** CodeT5** | 770M | 1 | 6.60 | 1.03 | 0.30 | 2.00 |
23+ | ** GPT-Neo** | 2.7B | 1 | 14.68 | 9.85 | 6.54 | 10.15 |
24+ | | | 5 | 19.89 | 13.19 | 9.90 | 13.87 |
25+ | ** GPT-2** | 0.1B | 1 | 5.64 | 6.93 | 4.37 | 6.16 |
26+ | | | 5 | 13.81 | 10.97 | 7.03 | 10.75 |
27+ | | 1.5B | 1 | 7.40 | 9.11 | 5.05 | 7.96 |
28+ | | | 5 | 16.86 | 13.84 | 9.01 | 13.48 |
29+ | ** GPT-3** | 175B | 1 | 0.57 | 0.65 | 0.21 | 0.55 |
30+ | ** StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
31+ | ** WizardCoder** | 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
5732| ** MoTCoder** | 15B | 1 | ** 33.80** | ** 19.70** | ** 11.09** | ** 20.80** |
58- | ** code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
59- | ** GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
33+ | ** text-davinci-002** | - | 1 | - | - | - | 7.48 |
34+ | ** code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
35+ | ** GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
36+
6037
6138** Performance on CodeContests**
6239| Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |
0 commit comments