How to properly configure timeouts and retries in Elasticsearch Python client?

[aymenkrifa]

Hello,

I'm currently using Elasticsearch Python version 7.17.12 in Python 3.10, and I want to eliminate the following read timeout error message. As shown in the following log, it saves the document in the second retry, but I want to make sure to increase the timeout so it doesn't fail, or if it fails print a logging message of retrying or something similar. I got confused with the different parameters of timeout and request_timeout and if they are in milliseconds or seconds

Here's my code:

client = Elasticsearch(
                hosts=self.host,
                use_ssl=True if self.ssl_certificate_path else False,
                verify_certs=False,
                ca_certs=self.ssl_certificate_path,
                ssl_show_warn=False,
                request_timeout=60000,
                max_retries=5,
                retry_on_timeout=True,
            )

and the index logic is:

client.index(
    index="index_name", body=object_to_save
)

2025-03-21T17:54:33.402+01:00 21-03-2025 16:54:33 - WARNING - POST https://XX.XXX.XX.XX:2000/index_name/_doc [status:N/A request:10.005s]
2025-03-21T17:54:33.402+01:00 Traceback (most recent call last):
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 468, in _make_request
2025-03-21T17:54:33.402+01:00     six.raise_from(e, None)
2025-03-21T17:54:33.402+01:00   File "<string>", line 3, in raise_from
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 463, in _make_request
2025-03-21T17:54:33.402+01:00     httplib_response = conn.getresponse()
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/http/client.py", line 1375, in getresponse
2025-03-21T17:54:33.402+01:00     response.begin()
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/http/client.py", line 318, in begin
2025-03-21T17:54:33.402+01:00     version, status, reason = self._read_status()
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/http/client.py", line 279, in _read_status
2025-03-21T17:54:33.402+01:00     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/socket.py", line 717, in readinto
2025-03-21T17:54:33.402+01:00     return self._sock.recv_into(b)
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/ssl.py", line 1307, in recv_into
2025-03-21T17:54:33.402+01:00     return self.read(nbytes, buffer)
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/ssl.py", line 1163, in read
2025-03-21T17:54:33.402+01:00     return self._sslobj.read(len, buffer)
2025-03-21T17:54:33.402+01:00 TimeoutError: The read operation timed out
2025-03-21T17:54:33.402+01:00 
2025-03-21T17:54:33.402+01:00 During handling of the above exception, another exception occurred:
2025-03-21T17:54:33.402+01:00 
2025-03-21T17:54:33.402+01:00 Traceback (most recent call last):
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 255, in perform_request
2025-03-21T17:54:33.402+01:00     response = self.pool.urlopen(
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 802, in urlopen
2025-03-21T17:54:33.402+01:00     retries = retries.increment(
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 527, in increment
2025-03-21T17:54:33.402+01:00     raise six.reraise(type(error), error, _stacktrace)
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
2025-03-21T17:54:33.402+01:00     raise value
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 716, in urlopen
2025-03-21T17:54:33.402+01:00     httplib_response = self._make_request(
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 470, in _make_request
2025-03-21T17:54:33.402+01:00     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2025-03-21T17:54:33.402+01:00   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
2025-03-21T17:54:33.402+01:00     raise ReadTimeoutError(
2025-03-21T17:54:33.402+01:00 urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='XX.XXX.XX.XX', port=2000): Read timed out. (read timeout=10)
2025-03-21T17:54:33.454+01:00 21-03-2025 16:54:33 - INFO - POST https://XX.XXX.XX.XX:2000/index_name/_doc [status:201 request:0.052s]

[gioboa]

These are for the latest version, I can't find the same page for v7.17 but I guess will be the same logic.
Based on this documentation request_timeout is a client side timeout and it's in second.
Here you can find info about server timeouts timeout master_timeout , these are using a unit of seconds.

[aymenkrifa]

Got it thank you for your help @gioboa!

So, does changing the request_timeout argument value to 60 instead of 60000 solve the issue? So I assume the 60000 value is too long, so it defaults to 10 seconds instead?

[aymenkrifa]

To test this, I set the request_timeout value to an extremely short duration (0.001) to reproduce the error.

Passing request_timeout to the class constructor doesn't have any effect, but passing it directly to the index method works as expected:

self.client.index(
    index=index,
    body=object_to_save,
    request_timeout=0.001,
)

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=0.001)

Is there a functional difference between setting request_timeout at the client instance level versus passing it directly to the index method?

[gioboa]

From my understanding the key difference lies in when and where the timeout is applied.

• Constructor Timeout: The timeout you set in the constructor (Elasticsearch(..., request_timeout=...)) becomes the default timeout used by the Transport class if no other timeout is specified. It sets a default value for future requests.

• Method Timeout (.index(..., request_timeout=...)): The timeout you set in the .index() method is passed directly to the API call's underlying perform_request method. This request_timeout parameter, when present, always takes precedence over the default timeout set on the client. It's a direct, per-request override.

[aymenkrifa]

Perfect, thank you!