SRU is a recurrent unit that can run over 10 times faster than cuDNN LSTM, without loss of accuracy tested on many tasks.
Average processing time of LSTM, conv2d and SRU, tested on GTX 1070
For example, the figures above and below present the processing time of a single mini-batch and the training time for sentence-level classification tasks. SRU achieves 10 to 16 times speed-up compared to LSTM, and operates as fast as (or faster than) word-level convolutional model (CNNs).
Training time (x-axis) vs valid accuracies (y-axis) on classification benchmarks
CuPy and pynvrtc needed to compile the CUDA code into a callable function at runtime.
- classification
- question answering on SQuAD
- language modelling on PTB
- machine translation
- speech recognition
- Tao Lei ([email protected])
- Yu Zhang ([email protected])