Return raw tokens for experimental IMDB dataset

[zhangguanheng66]

Return raw text data.

Return tokens

import torchtext
from torchtext.experimental.datasets import IMDB
vocab = lambda x: x
train_dataset, test_dataset = IMDB(vocab=vocab)

Return raw text

import torchtext
from torchtext.experimental.datasets import IMDB
from torchtext.data.utils import get_tokenizer
vocab = lambda x: x
tokenizer = get_tokenizer('empty_tokenizer')
train_dataset, test_dataset = IMDB(vocab=vocab, tokenizer=tokenizer)

[zhangguanheng66]

fix #690
@bentrevett

[cpuhrsch]

The assumption here is that if vocab is not a callable it'll be an instance of Vocab? We could add a type assertion here at the beginning.

We could also create a version that accepts a callable and then pass in the callable lambda x: vocab[x].

[cpuhrsch]

In which cases will this conversion not succeed? Using try except is a pretty big hammer. ValueError could be raised by many functions.

[bentrevett]

This will fail when tokens is a list of string(s), e.g. using an "empty" vocab (vocab = lambda x : x) and a tokenizer to get the data as a list of strings or using the empty vocab with the new empty_tokenizer when you want your data as a string, untokenized.

[zhangguanheng66]

merged in #701