Torchtext datasets failed during multi-processing due to lambda functions usage

[parmeet]

Duplicate Issue pytorch/data#397

Issue

Currently, torchtext datasets make use of lambda functions that creates issue for MultiProcessing with DataLoader as pickle doesn't support serialization with lambda functions.

Tracking the issue here as well for visibility

Suggested Resolution

Remove usage of lambda functions

cc: @NivekT , @NicolasHug

[NicolasHug]

Thanks @parmeet for keeping track of this!

I think there are still instances of the use of lambdas in the tutorials:

text/examples/tutorials/sst2_classification_non_distributed.py

Line 98 in dfb53af

train_datapipe = train_datapipe.map(lambda x: (text_transform(x[0]), x[1]))

The DataLoader doesn't use multi-processing for now

text/examples/tutorials/sst2_classification_non_distributed.py

Line 101 in dfb53af

train_dataloader = DataLoader(train_datapipe, batch_size=None)

so it shouldn't be a problem in the tutorial script itself.

But in general, users might start tweaking the tutorial and set num_workers=k. This might cause problems because of the use of lambdas, but also with the RNG and shuffling issues that I raised in pytorch/data#302.

Since these datasets are released already, I was wondering if torchtext had a plan to document these pitfalls before the next release?

[parmeet]

Thanks @NicolasHug, sure will update the tutorial.

Regarding some of the pitfalls, I think it's a good idea to document. Will look into it and see to create a tracking issue that can be referenced in release notes or something :).