[WIP] add unicode generation to IWSLT tests

[erip]

Addresses #1607

[parmeet]

Does this sounds familiar https://app.circleci.com/pipelines/github/pytorch/text/4926/workflows/b22fa7de-be18-4fde-af6e-73d86731fb80/jobs/164125 ?

[parmeet]

Does this sounds familiar https://app.circleci.com/pipelines/github/pytorch/text/4926/workflows/b22fa7de-be18-4fde-af6e-73d86731fb80/jobs/164125 ?

@erip This issue seems random. It didn't happen in our latest unit-testing on the release branch #1613. I am not sure why it is the case. Do you think it has anything to do with XML not well formed as you propose the fix in this PR?

[erip]

Sorry -- I must've missed this notification. It's unclear, but it seems like it would be a consistent error since the closing tag always ends like ">>"... Hmm

[Nayef211]

Hey @erip. I just wanted to touch base on this PR again to see what the blockers and next steps might be? I believe this is the only thing left before we can close out #1493

[erip]

Sorry -- the semester has ramped up a bit for me in the last couple of weeks. 😄 I suspect (but haven't actually tested) that adding lxml support for reading would address this problem (by way of XMLParser's recover kwarg . I'm a bit hesitant about that approach because it adds a runtime dependency to address an issue that tests would introduce since ... we don't want users to silently recover from their own broken XML.

[Nayef211]

Sorry -- the semester has ramped up a bit for me in the last couple of weeks. 😄 I suspect (but haven't actually tested) that adding lxml support for reading would address this problem (by way of XMLParser's recover kwarg . I'm a bit hesitant about that approach because it adds a runtime dependency to address an issue that tests would introduce since ... we don't want users to silently recover from their own broken XML.

Gotcha thanks for the context. I can take over investigating how to fix this test. I was also wondering why we have "</doc" in the xml_tags list even though its a closing tag (https://github.com/pytorch/text/blob/main/test/datasets/test_iwslt2016.py#L42)? Wouldn't this lead to an incorrectly formed XML file?

[Nayef211]

Gotcha thanks for the context. I can take over investigating how to fix this test. I was also wondering why we have "</doc" in the xml_tags list even though its a closing tag (https://github.com/pytorch/text/blob/main/test/datasets/test_iwslt2016.py#L42)? Wouldn't this lead to an incorrectly formed XML file?

I added a new PR (#1642) that removes the closing tag from the xml tags list. Lmk what you guys think @erip @parmeet

[Nayef211]

Closing this now that #1642 has been merged