Skip to content

Commit 7e5e182

Browse files
PringledPringled
authored
docs: Added training plot, added more training results (MinishLab#189)
* Updated results * Updated results
1 parent c6ce322 commit 7e5e182

File tree

3 files changed

+43
-26
lines changed

3 files changed

+43
-26
lines changed

‎assets/images/training_speed_vs_score.png

211 KB
Loading

‎model2vec/train/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,3 +106,7 @@ The core functionality of the `StaticModelForClassification` is contained in a c
106106
* `fit`: contains all the lightning-related fitting logic.
107107

108108
The training of the model is done in a `lighting.LightningModule`, which can be modified but is very basic.
109+
110+
# Results
111+
112+
We ran extensive benchmarks where we compared our model to several well known architectures. The results can be found in the [training results](https://github.com/MinishLab/model2vec/tree/main/results#training-results) documentation.

‎results/README.md

Lines changed: 39 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -70,43 +70,56 @@ As can be seen, [potion-retrieval-32M](https://huggingface.co/minishlab/potion-r
7070

7171
The main results for Model2Vec training are outlined in this section.
7272

73-
We compare three different architectures:
73+
We compare five different architectures for our main results:
7474
- `model2vec + logreg`: A model2vec model with a scikit-learn `LogisticRegressionCV` on top.
7575
- `model2vec full finetune`: A model2vec classifier with the full model finetuned. This uses our `StaticModelForClassification`.
76+
- `tfidf`: A TF-IDF model with a scikit-learn `LogisticRegressionCV` on top.
7677
- `setfit`: A [SetFit](https://github.com/huggingface/setfit/tree/main) model trained using [all-minilm-l6-v2](sentence-transformers/all-MiniLM-L6-v2) as a base model.
78+
- `bge-base + logreg`: A [BGE-base](https://huggingface.co/BAAI/bge-base-en-v1.5) encoder model with a scikit-learn `LogisticRegressionCV` on top.
7779

7880
We use 14 classification datasets, using 1000 examples from the train set, and the full test set. No parameters were tuned on any validation set. All datasets were taken from the [Setfit organization on Hugging Face](https://huggingface.co/datasets/SetFit).
7981

80-
| dataset | model2vec + logreg | model2vec full finetune | setfit |
81-
|:---------------------------|----------------------------------------------:|---------------------------------------:|-------------------------------------------------:|
82-
| 20_newgroups | 56.24 | 57.94 | 61.29 |
83-
| ade | 79.2 | 79.68 | 83.05 |
84-
| ag_news | 86.7 | 87.2 | 88.01 |
85-
| amazon_counterfactual | 90.96 | 91.93 | 95.51 |
86-
| bbc | 95.8 | 97.21 | 96.6 |
87-
| emotion | 65.57 | 67.11 | 72.86 |
88-
| enron_spam | 96.4 | 96.85 | 97.45 |
89-
| hatespeech_offensive | 83.54 | 85.61 | 87.69 |
90-
| imdb | 85.34 | 85.59 | 86 |
91-
| massive_scenario | 82.86 | 84.42 | 83.54 |
92-
| senteval_cr | 77.03 | 79.47 | 86.15 |
93-
| sst5 | 32.34 | 37.95 | 42.31 |
94-
| student | 83.2 | 85.02 | 89.62 |
95-
| subj | 89.2 | 89.85 | 93.8 |
96-
| tweet_sentiment_extraction | 64.96 | 62.65 | 75.15 |
97-
98-
| | logreg | full finetune | setfit
99-
|:---------------------------|-----------:|---------------:|-------:|
100-
| average | 77.9 | 79.2 | 82.6 |
82+
| dataset | tfidf | model2vec + logreg | model2vec full finetune | setfit | bge-base + logreg |
83+
|:---------------------------|--------:|---------------------:|--------------------------:|---------:|--------------------:|
84+
| 20_newgroups | 50.71 | 56.24 | 57.94 | 61.29 | 67.39 |
85+
| ade | 71.46 | 79.20 | 79.68 | 83.05 | 86.12 |
86+
| ag_news | 81.68 | 86.70 | 87.20 | 88.01 | 88.95 |
87+
| amazon_counterfactual | 85.18 | 90.96 | 91.93 | 95.51 | 92.74 |
88+
| bbc | 95.09 | 95.80 | 97.21 | 96.60 | 97.50 |
89+
| emotion | 59.28 | 65.57 | 67.11 | 72.86 | 65.63 |
90+
| enron_spam | 96.00 | 96.40 | 96.85 | 97.45 | 97.30 |
91+
| hatespeech_offensive | 66.45 | 83.54 | 85.61 | 87.69 | 84.92 |
92+
| imdb | 80.44 | 85.34 | 85.59 | 86.00 | 92.25 |
93+
| massive_scenario | 77.26 | 82.86 | 84.42 | 83.54 | 87.07 |
94+
| senteval_cr | 65.61 | 77.03 | 79.47 | 86.15 | 90.53 |
95+
| sst5 | 18.52 | 32.34 | 37.95 | 42.31 | 38.49 |
96+
| student | 74.16 | 83.20 | 85.02 | 89.62 | 89.71 |
97+
| subj | 86.39 | 89.20 | 89.85 | 93.80 | 94.55 |
98+
| tweet_sentiment_extraction | 53.20 | 64.96 | 62.65 | 75.15 | 69.48 |
99+
100+
101+
| | tfidf | model2vec + logreg | model2vec full finetune | setfit | bge-base + logreg |
102+
|:--------|--------:|---------------------:|--------------------------:|---------:|--------------------:|
103+
| average | 70.8 | 78.0 | 79.2 | 82.6 | 82.8 |
104+
105+
106+
101107

102108
As can be seen see, full fine-tuning brings modest performance improvements in some cases, but very large ones in other cases, leading to a pretty large increase in average score. Our advice is to test both if you can use `potion-base-32m`, and to use full fine-tuning if you are starting from another base model.
103109

104-
The speed difference between model2vec and setfit is immense, with the full finetune being 35x faster than a setfit based on `all-minilm-l6-v2` on CPU.
110+
The speed difference between model2vec and the other models is immeense, with the full finetune being 35x faster than a setfit based on `all-minilm-l6-v2` on CPU and 200x faster than the`bge-base` transformer model.
105111

106-
| | logreg | full finetune | setfit
107-
|:---------------------------|-----------:|---------------:|-------:|
108-
| samples / second | 17925 | 24744 | 716 |
109112

113+
| | tfidf | model2vec + logreg | model2vec full finetune | setfit | bge-base + logreg |
114+
|:-----------------|--------:|---------------------:|--------------------------:|---------:|--------------------:|
115+
| samples / second | 108434 | 17925 | 24744 | 716 | 118 |
116+
117+
118+
The figure below shows the relationship between the number of sentences per second and the average training score, where we've included more transformer-based models for comparison.
119+
120+
| ![Description](../assets/images/training_speed_vs_score.png) |
121+
|:--:|
122+
|*Figure: The average training score plotted against sentences per second (log scale).*|
110123

111124

112125
## Ablations

0 commit comments

Comments
 (0)