TREC dataset

[bmccann]

import torch
from torchtext import data
from torchtext import datasets
inputs = data.Field(lower=True, include_lengths=True, batch_first=True)
answers = data.Field(sequential=False)
print('Generating train, dev, test splits')
train, test = datasets.TREC.splits(inputs, answers)
print(train[0].text, train[0].label)

yields

Generating train, dev, test splits
['how', 'did', 'serfdom', 'develop', 'in', 'and', 'then', 'leave', 'russia', '?'] DESC

When fine_grained=True:

Generating train, dev, test splits
['how', 'did', 'serfdom', 'develop', 'in', 'and', 'then', 'leave', 'russia', '?'] DESC:manner

Let me know if there is anything else you'd like to see. I'll leave the WIP until I hear
back.

[jekbradbury]

Does the dataset contain corrupted UTF-8? If not I’d rather just load it with Python rather than adding another dependency? (If it does then this is probably fine, although a built-in error strategy might also work)