Description
Elasticsearch Version
version: 8.17.7, Build: docker/7e0ac58561b31b851200221380b3a6c5243d5b37/2025-05-21T22:08:05.149207602Z, JVM: 23
Installed Plugins
cluster.stats.source_modes, data_stream.auto_sharding, data_stream.lifecycle.global_retention, data_stream.rollover.lazy, desired_node.version_deprecated, esql.agg_values, esql.async_query, esql.base64_decode_encode, esql.casting_operator, esql.counter_types, esql.disable_nullable_opts, esql.from_options, esql.metadata_fields, esql.metrics_counter_fields, esql.mv_ordering_sorted_ascending, esql.mv_sort, esql.resolve_fields_api, esql.spatial_points_from_source, esql.spatial_shapes, esql.st_centroid_agg, esql.st_contains_within, esql.st_disjoint, esql.st_intersects, esql.st_x_y, esql.string_literal_auto_casting, esql.string_literal_auto_casting_extended, esql.timespan_abbreviations, features_supported, file_settings, flattened.ignore_above_support, geoip.downloader.database.configuration, get_database_configuration_action.multi_node, health.dsl.info, health.extended_repository_indicator, knn_retriever_supported, license-trial-independent-version, logsdb_telemetry, logsdb_telemetry_stats, mapper.boolean_dimension, mapper.flattened.ignore_above_with_arrays_support, mapper.ignore_above_index_level_setting, mapper.index_sorting_on_nested, mapper.keyword_dimension_ignore_above, mapper.keyword_normalizer_synthetic_source, mapper.pass_through_priority, mapper.query_index_mode, mapper.range.null_values_off_by_one_fix, mapper.segment_level_fields_stats, mapper.source.synthetic_source_copy_to_fix, mapper.source.synthetic_source_copy_to_inside_objects_fix, mapper.source.synthetic_source_fallback, mapper.source.synthetic_source_stored_fields_advance_fix, mapper.source.synthetic_source_with_copy_to_and_doc_values_false, mapper.subobjects_auto, mapper.subobjects_auto_fixes, mapper.synthetic_source_keep, mapper.track_ignored_source, mapper.vectors.bbq, mapper.vectors.bit_vectors, mapper.vectors.int4_quantization, put_database_configuration_action.ipinfo, query_rule_list_types, query_rule_retriever_supported, query_rules.test, random_reranker_retriever_supported, repositories.supports_usage_stats, rest.capabilities_action, rest.local_only_capabilities, retrievers_supported, routing.boolean_routing_path, routing.multi_value_routing_path, rrf_retriever_composition_supported, rrf_retriever_supported, script.hamming, script.term_stats, search.vectors.k_param_supported, security.migration_framework, security.role_mapping_cleanup, security.roles_metadata_flattened, semantic_text.default_elser_2, semantic_text.search_inference_id, simulate.component.template.substitutions, simulate.index.template.substitutions, simulate.mapping.addition, simulate.mapping.validation, simulate.mapping.validation.templates, simulate.support.non.template.mapping, slm.interval_schedule, snapshot.repository_verify_integrity, standard_retriever_supported, stats.include_disk_thresholds, text_similarity_reranker_retriever_composition_supported, text_similarity_reranker_retriever_supported, tsdb.ts_routing_hash_doc_value_parse_byte_ref, unified_highlighter_matched_fields, usage.data_tiers.precalculate_stats
Java Version
bundled
OS Version
Linux elasticsearch-es-eck-set-2-3 6.1.0-32-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06) x86_64 x86_64 x86_64 GNU/Linux
Problem Description
I hope this is not a duplicate, i found many geoip issues when trying to see if it was already reported, but i did not find anything matching exactly
When running elasticsearch in an ipv6 only kubernetes via eck cloud opperator. it seems that elasticsearch fail to download the geoip plugin databases.
log from elasticsearch
elasticsearch-es-eck-set-2-3 elasticsearch {"@timestamp":"2025-06-23T08:21:49.439Z", "log.level":"ERROR", "message":"exception during geoip databases update", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-eck-set-2-3][generic][T#10]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"8IteDtAoSHu5llN0YQig5g","elasticsearch.node.id":"jJ764HAoSICcMgBwkuoaiQ","elasticsearch.node.name":"elasticsearch-es-eck-set-2-3","elasticsearch.cluster.name":"elasticsearch","error.type":"java.net.SocketException","error.message":"Network is unreachable","error.stack_trace":"java.net.SocketException: Network is unreachable\n\tat java.base/sun.nio.ch.Net.connect0(Native Method)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:589)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:578)\n\tat java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583)\n\tat java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)\n\tat java.base/java.net.Socket.connect(Socket.java:760)\n\tat java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)\n\tat java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1228)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1114)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1676)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1600)\n\tat java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)\n\tat java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:97)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:157)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:91)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:78)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:74)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:174)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:157)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:300)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:166)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:65)\n\tat [email protected]/org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:35)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\n"}
when testing manually with curl inside the conainer it downloads without any issues.
note that i use NAT64 to reach ipv4 only resources on the internet.
elasticsearch@elasticsearch-es-eck-set-2-3:/tmp$ getent hosts updates.maxmind.com
2606:4700:7::a29f:8716 updates.maxmind.com
2606:4700:7::a29f:8616 updates.maxmind.com
elasticsearch@elasticsearch-es-eck-set-2-3:/tmp$ getent hosts geoip.elastic.co
64:ff9b::2248:efb7 geoip.elastic.co
elasticsearch@elasticsearch-es-eck-set-2-3:/tmp$ curl -O https://geoip.elastic.co/v1/database/GeoLite2-City.mmdb
elasticsearch@elasticsearch-es-eck-set-2-3:/tmp$ curl -O https://geoip.elastic.co/v1/database/GeoLite2-ASN.mmdb
elasticsearch@iteam-elasticsearch-es-iteam-eck-set-2-3:/tmp$ ls -lha
total 24K
drwxrwsrwx 4 root elasticsearch 4.0K Jun 23 08:43 .
drwxr-xr-x 1 root root 4.0K Jun 23 07:58 ..
-rw-r--r-- 1 elasticsearch elasticsearch 419 Jun 23 08:43 GeoLite2-ASN.mmdb
-rw-r--r-- 1 elasticsearch elasticsearch 419 Jun 23 08:43 GeoLite2-City.mmdb
the geoip downloader must do something else to trigger this error. perhaps it tries to use ipv4 litterals directly?
kinds regards
Ronny Aasen
Steps to Reproduce
- ipv6 only kubernetes cluster with DNS64/NAT64
- install eck
- install elasticsearch via eck, include ingest.geoip.downloader.eager.download: true
- observe the logs.
- a test via console show "_geoip_database_unavailable_GeoLite2-City.mmdb"
Logs (if relevant)
elasticsearch-es-eck-set-2-3 elasticsearch {"@timestamp":"2025-06-23T08:21:49.439Z", "log.level":"ERROR", "message":"exception during geoip databases update", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-eck-set-2-3][generic][T#10]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"8IteDtAoSHu5llN0YQig5g","elasticsearch.node.id":"jJ764HAoSICcMgBwkuoaiQ","elasticsearch.node.name":"elasticsearch-es-eck-set-2-3","elasticsearch.cluster.name":"elasticsearch","error.type":"java.net.SocketException","error.message":"Network is unreachable","error.stack_trace":"java.net.SocketException: Network is unreachable\n\tat java.base/sun.nio.ch.Net.connect0(Native Method)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:589)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:578)\n\tat java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583)\n\tat java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)\n\tat java.base/java.net.Socket.connect(Socket.java:760)\n\tat java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)\n\tat java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1228)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1114)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1676)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1600)\n\tat java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)\n\tat java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:97)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:157)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:91)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:78)\n\tat [email protected]/org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:74)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:174)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:157)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:300)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:166)\n\tat [email protected]/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:65)\n\tat [email protected]/org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:35)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\n"}