Skip to content

Multi30k mocked testing #1554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Feb 2, 2022
Merged

Multi30k mocked testing #1554

merged 4 commits into from
Feb 2, 2022

Conversation

parmeet
Copy link
Contributor

@parmeet parmeet commented Jan 30, 2022

Reference issue #1493

erip
erip reviewed Jan 30, 2022
View reviewed changes
test/datasets/test_multi30k.py
Comment on lines +27 to +31
with open(txt_file, "w") as f:
for i in range(5):
rand_string = " ".join(
random.choice(string.ascii_letters) for i in range(seed)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought: since all of our datasets are utf-8 files, does it make sense to write unicode strings to make sure we don't have lingering bugs from default encodings when opening files? Maybe this is overkill, but it's been a big source of bugs when I did mostly windows development.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, agreed. I think it is a good suggestion!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will keep it as a follow-up item as generating random UTF-8 string is not trivial.

Nayef211
Nayef211 approved these changes Jan 31, 2022
View reviewed changes
Copy link
Contributor

@Nayef211 Nayef211 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Left some nit comments. Also think @erip has a valid point about writing unicode strings when we mock data. Should we do this for all our tests moving forward?

test/datasets/test_multi30k.py Outdated
rand_string = " ".join(
random.choice(string.ascii_letters) for i in range(seed)
)
content = f'{rand_string}\n'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we use double quotes here instead of single quotes

@Nayef211 Nayef211 mentioned this pull request Jan 31, 2022
Closed
27 tasks
@parmeet
Copy link
Contributor Author

parmeet commented Feb 2, 2022

Overall LGTM. Left some nit comments. Also think @erip has a valid point about writing unicode strings when we mock data. Should we do this for all our tests moving forward?

Yes, let's add a follow-up item here or even better if we can also track this in the issue #1493 in general.

@parmeet parmeet merged commit 69825a1 into pytorch:main Feb 2, 2022
@parmeet parmeet deleted the multi30k_test branch February 2, 2022 22:37
@Nayef211
Copy link
Contributor

Nayef211 commented Feb 3, 2022

Yes, let's add a follow-up item here or even better if we can also track this in the issue #1493 in general.

Added this as a follow up item to the GH issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erip erip erip left review comments

@Nayef211 Nayef211 Nayef211 approved these changes

Assignees
No one assigned
Labels
ciflow/default cla signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@parmeet @Nayef211 @erip @facebook-github-bot