Evaluate newer models with GLUE

[mattdangerw]

We have added a few more models, deberta, f_net, and albert, that are meant for classification and could do with GLUE evaluation.

Let's run some testing on them and make sure they are performance in the right ballpark!

[mattdangerw]

Ideally, we could figure out a way to just have a lot of automation here. E.g. a single command we can run to get a clue score for a given classification model on GCP.

[susnato]

Hi @chenmoneygithub forgive me for asking but have you started working on this issue? If not then I would like to work on this one. Is it possible to do that?

[chenmoneygithub]

@susnato Thanks for your interest! There are two parts -

1. Evaluate these models with the code in examples/glue_benchmark on GCP, and it requires some manual hyper tuning.
2. Write a script that can be run as nightly testing, but here we may want to focus only on light-weight tasks. To be specific, glue/mnli is not suitable for automatic testing due to its data volume.

We don't have GCP credits for contributors right now, but I think we can start with writing a script similar to this link, and we can use glue/mrpc and glue/stsb, and report dev scores along with time elapse. Due to the small size, you can use the free GPU resource on colab to achieve it. Are you interested in it? thanks ahead!

[susnato]

@chenmoneygithub Yes I am very interested in it. If possible, please assign it to me.