Empty ES response on search query returning shards 0

[hawk-roman-rey]

I am using elasticsearch py client in my real time application.

The application performs search query aggregations like "sum of field settledAmount in last 10 days".

Most of queries work without problems. However a tiny percent fail silently returning the following.

    {
       "took":0,
       "timed_out":false,
       "_shards":{
          "total":0,
          "successful":0,
          "skipped":0,
          "failed":0
       },
       "hits":{
          "total":{
             "value":0,
             "relation":"eq"
          },
          "max_score":0.0,
          "hits":[
             
          ]
       }
    }

When the same query is repeated the response is correct containing the aggregations field and the expected value.

  {
     "took" : 191,
     "timed_out" : false,
     "_shards" : {
       "total" : 756,
       "successful" : 756,
       "skipped" : 632,
       "failed" : 0
     },
     "hits" : {
       "total" : {
         "value" : 0,
         "relation" : "eq"
       },
       "max_score" : null,
       "hits" : [ ]
     },
     "aggregations" : {
       "total_settled_amount" : {
         "value" : 0.0
       },
       "unique_customer_count" : {
         "value" : 0
       }
     }
   }

I added some retries and it seems to be related with a load problem, since retrying the query without a significant delay (1s) will end in the exact same wrong result.

This is a query example

    GET cases-*/_search
    {
      "from": 0,
      "size": 1000,
      "query": {
        "bool": {
          "filter": [
            {
              "range": {
                "transactionAnnouncedEvent.transaction.processedAt": {
                  "from": "2024-08-29T00:00:00.000Z",
                  "to": "2024-08-29T13:46:49.000Z",
                  "include_lower": true,
                  "include_upper": true,
                  "boost": 1
                }
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "term": {
                      "field1": {
                        "value": "abc",
                        "boost": 1
                      }
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "boost": 1
              }
            }
          ],
          "adjust_pure_negative": true,
          "boost": 1
        }
      },
      "_source": {
        "includes": [
          "caseId"
        ],
        "excludes": []
      },
      "aggs": {
        "total_settled_amount": {
          "sum": {
            "field": "settledAmount"
          }
        }
      }
    }

What might be the reason of elasticsearch returning such "empty response" instead of a "429 status code?"

[pquentin]

Hello @hawk-roman-rey, this is quite surprising. I don't know how this could happen and I can't help you as I only maintain the Python client and this question is about the server. I would suggest asking in https://discuss.elastic.co/ Thanks for your understanding.

[hawk-roman-rey]

I fully understand it, what would you suggest to debug this issue on elasticsearch-py side? Are there any ways to enable debug logs or any hacks you could think of?

I've posted the question in the elastic forum you suggested, will comment here if something helps

[hawk-roman-rey]

Something I forgot mentioning:

• Some java applications perform the same kind of queries to the same ES cluster and don't show such issue

[pquentin]

For reference, the Discuss thread is https://discuss.elastic.co/t/elasticsearch-search-query-returning-empty-response-using-elasticsearch-python-api/365983.

One thing you could do on the client side is enabling debug logging to be certain that you don't have a bug in your code where sometimes you're not asking for the aggregation. See https://elastic-transport-python.readthedocs.io/en/latest/logging.html for instructions.

[pquentin]

Closing this in favor of the Discuss thread. Thank you!