ElasticsearchWarning quits query in pytest

[mortenb-buypass]

elasticsearch = 8.17.0

When the elasticsearch.ElasticsearchWarning triggers the response returns None.

A warning should just print and continue.

The workaround it to add a limit to everything

        try:
            response = self.client.esql.query(
                query=query, format='csv'
            )
        except (elasticsearch.exceptions.BadRequestError) as e:
            log.error(e)
            return None
        except (KeyboardInterrupt, SystemExit):
            log.error("interrupted")
            return None
        except (elasticsearch.ElasticsearchWarning) as e:
            log.warning(e)
            pass
        return pd.read_csv(StringIO(response.body))

20250115125410.945|WARNING|esql.py:192|No limit defined, adding default limit of [1000]
:
>       log.info(df.to_string(justify='left', index=False))
E       AttributeError: 'NoneType' object has no attribute 'to_string'

[pquentin]

Hello, and thanks for raising this issue. However, warnings raised by the library don't raise exceptions:

elasticsearch-py/elasticsearch/__init__.py

Lines 71 to 73 in e4a9563

	# Only raise one warning per deprecation message so as not
	# to spam up the user if the same action is done multiple times.
	warnings.simplefilter("default", category=ElasticsearchWarning, append=True)

(If it were error instead of default, it would raise an exception. See https://docs.python.org/3/library/warnings.html#the-warnings-filter for more details.)

The line responsible for the exception is log.info(df.to_string(justify='left', index=False)). Since df is None, you can't call to_string on it. However, this is your code, not code from elasticsearch-py. You can add | LIMIT 1000 to your query to confirm that the warning is not the issue.

[mortenb-buypass]

Must be something in pytest, direct it works fine, thanks for the help, sorry to have bugged You :-)

#!/usr/bin/env python3
import elasticsearch
query = r'FROM bps-trace-* | WHERE @timestamp >= "2025-01-15T00:59:24.521Z" AND @timestamp <= "2025-01-15T12:59:24.523Z" | WHERE bps.application like """*trk*test1""" | KEEP bps.application | STATS aggs = count(bps.application) BY bps.application | SORT aggs DESC'
e = elasticsearch.Elasticsearch("http://kibana.test.buypass.no:9200", timeout=60)
resp = e.esql.query(
    query=query, 
    format='csv'
)
print(resp.body)

python.exe .\tools\t1.py
C:\dist\trk-fullstack-test\tools\t1.py:4: DeprecationWarning: The 'timeout' parameter is deprecated in favor of 'request_timeout'
  e = elasticsearch.Elasticsearch("http://kibana.test.buypass.no:9200", timeout=60)
C:\dist\trk-fullstack-test\tools\t1.py:5: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  resp = e.esql.query(
aggs,bps.application
4510,id-pro-trk-keycloak-trk-keycloak-server-test1
572,id-pro-trk-trk-phone-service-test1
426,id-pro-trk-trk-otp-service-test1
292,id-pro-trk-trk-fido2-service-test1
260,id-pro-trk-trk-passport-service-test1
223,id-pro-trk-trk-user-portal-web-test1
201,id-pro-trk-trk-freg-update-service-test1
171,id-pro-trk-trk-operator-portal-web-test1
142,id-pro-trk-trk-scim-api-test1
22,id-pro-trk-trk-sms-gateway-test1
16,id-pro-trk-trk-freg-gateway-test1

[pquentin]

Right, pytest can be configured to raise errors on warnings so that you don't miss them. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html for more details.