Add a timeout to the url requests #207
Conversation
Timeout of 2 seconds.
tldr.py
Outdated
| @@ -160,6 +160,7 @@ def get_page_for_platform( | |||
| try: | |||
| data = urlopen( | |||
| Request(page_url, headers=REQUEST_HEADERS), | |||
| timeout = 2, | |||
There was a problem hiding this comment.
| timeout = 2, | |
| timeout=2, |
There was a problem hiding this comment.
Is two seconds long enough?
There was a problem hiding this comment.
Is two seconds long enough?
I think so, but i wouldn't care if it was set to 10 seconds or something like that. The problem that i saw was that, if no timeout was specified, there were some occasions where the program would get stuck for more than a minute until throwing a timeout error. However I believe that 2 seconds are more than enough.
There was a problem hiding this comment.
I've bumped this up to 10 seconds as I feel like while 2 is probably fine, better to be a touch conservative and hopefully avoid any people complaining about things "suddenly" breaking on them due to timeouts, and this still at least more rapidly gets this program to crash out much more quickly.
tldr.py
Outdated
| except TimeoutError: | ||
| continue |
There was a problem hiding this comment.
I'm not sure we should be skipping this, in that we don't skip if we get a 500 HTTPError. Something about the upstream is broken in a way that we cannot connect to it, and I don't think we should continue on and potentially serve the wrong page.
There was a problem hiding this comment.
I've read some more info about that and I've noticed that in fact a URLError (and not a TimeoutError) is raised when the timeout expires. Consequently I've deleted that part of the code and committed it. The behavior is that whenever a request is not answered an error message is shown to the user. Sorry for the confusion.
Co-authored-by: Matthew Peveler <[email protected]>
When the timeout expires, a URLError is thrown (not a TimeoutError).
Just added a timeout to the url request so when it's not answered it doesn't wait a long time before throwing an error.