How to upgrade self-hosted Sentry from v21.5.0 to latest with minimal downtime on production environment?

[siva-mareedu]

Self-Hosted Version

21.5.0

CPU Architecture

x86_64

Docker Version

20.10.21

Docker Compose Version

1.26.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

2. cd /opt/sentry
3. git checkout main; git pull
4. git checkout local_sentry;
5. git merge --no-ff 21.5.0 -m "Merge tag '21.5.0' into local_sentry"
6. Fixed the Conflicts in .env [keeping only latest changes of images]
7. docker-compose down;
8. docker-compose up -d
9. docker-compose ps

Events are not flowing after this;

Expected Result

We are running a self-hosted Sentry (v21.4.1) on an EC2 instance which is deployed around 4 years ago. We want to upgrade to the latest version without losing the existing events in the system but we’re facing issues due to the large version gap.

We've Tried:: We attempted incremental upgrades (one version at a time). However, this approach is very time-consuming, and during one upgrade from v21.4.1 to v21.5.0, we experienced a 3-hour delay in event processing. The relay container showed repeated authentication errors like the following:

2025-06-30T09:57:29Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

2025-06-30T09:57:47Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

We verified that communication is working fine from relay and web manually through performing curl inside relay containers, but still throwing above error.
Although the issue eventually resolved itself, we are concerned that this will repeat in each upgrade step with causing long and repeated downtimes, which is not acceptable for our production system.

What We’re Looking For:: We’re looking for a safe and efficient way to upgrade from v21.4.1 to the latest self-hosted Sentry version (e.g. 25.x) with minimal downtime and no data loss.

We’re considering two approaches:

1. Direct Upgrade: Is it possible to upgrade directly from v21.4.1 → latest? What are the potential issues or migration steps involved?
2. Fresh Install + Data Migration: Can we migrate existing data (projects, issues, events, users, etc.) to newly created self hosted sentry instance ?? Are there recommended ways to achieve this using tools or scripts ?

Additional Details

Environment: PROD (EC2 instance with self hosted sentry deployment using docker-compose.yml).
Downtime tolerance: Minimal downtime preferred (a few minutes acceptable).
We are okay with either method (upgrade in-place or fresh install), as long as the existing production data is preserved and system reliability is maintained.

Any guidance, best practices or documentation links would be much appreciated.

Thank you !!

Actual Result

Got below errors during update to 21.5.0 from 21.4.1.
2025-06-30T09:57:29Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

2025-06-30T09:57:47Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

We want to upgrade our sentry to latest versions without losing data. Any guidance, best practices or documentation links would be much appreciated.

Event ID

No response

[aldy505]

Direct Upgrade: Is it possible to upgrade directly from v21.4.1 → latest? What are the potential issues or migration steps involved?

No. You will need to go through the hard stops one by one. So from your current version, it should be v21.5.0 -> v21.6.3 -> v23.6.2 -> v23.11.0 -> v24.8.0 -> v25.5.1. The hard stops usually have some kind of breaking changes in database migrations.

Fresh Install + Data Migration: Can we migrate existing data (projects, issues, events, users, etc.) to newly created self hosted sentry instance ?? Are there recommended ways to achieve this using tools or scripts ?

I wouldn't recommend this method if you don't want to lose existing data (event, issues). If it's okay to lose that, and keep projects & users data, you can use the backup & restore script. See the "Partial JSON backup" here https://develop.sentry.dev/self-hosted/backup/

Environment: PROD (EC2 instance with self hosted sentry deployment using docker-compose.yml).
Downtime tolerance: Minimal downtime preferred (a few minutes acceptable).
We are okay with either method (upgrade in-place or fresh install), as long as the existing production data is preserved and system reliability is maintained.

I would recommend putting another Relay before your Sentry instance for ingestion-only endpoints. See https://docs.sentry.io/product/relay/

Therefore, the events would be kept on that Relay (which usually 1 hour worth of events), and it will be processed at a later time when your Docker Compose is all up and running. This is a bit more complex, but you can see the overview here: https://github.com/getsentry/sentry-docs/pull/13893/files

2025-06-30T09:57:47Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

I believe this indicates another error, you need to look on your web container logs.

[siva-mareedu]

Thank you very much @aldy505 for your explanation.
As you suggested, we have initiated the upgrade process with hard stops [21.5.0 -> 21.6.3] in POC setup. We have taken the changes from 21.6.3 and manually resolved the merge conflicts in .env and docker-compose.yml before proceeding to execute the ./install.sh script.
During the installations, we encountered an issue related to the Debian registry. This was resolved by following the workaround described in #3650.

After resolving the issue, we re-ran the install.sh script and it completed successfully. All docker containers are now up and running.
We then verified the logs for relay by testing live event data through a script and no events are pushing to sentry. We received following errors in relay container.

ERROR: authentication encountered error: could not send request to upstream
caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

I've attached the logs of relay, web, zookeeper, kafka for more info. please find below logs of each container:

[relay]
2025-07-08T12:09:27Z [relay_server::actors::upstream] DEBUG: sending register challenge response
2025-07-08T12:09:27Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)
2025-07-08T12:09:27Z [relay_server::actors::upstream] DEBUG: scheduling authentication retry in 60 seconds
2025-07-08T12:05:26Z [relay_server::actors::upstream] DEBUG: sending register challenge response
2025-07-08T12:05:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)
2025-07-08T12:05:26Z [relay_server::actors::upstream] DEBUG: scheduling authentication retry in 60 seconds
2025-07-08T12:06:26Z [relay_server::actors::upstream] INFO: registering with upstream (http://web:9000/)
2025-07-08T12:06:26Z [relay_server::actors::upstream] DEBUG: got register challenge (token = eyJ0aW1lc3RhbXAiOjE3NTE5NzYzODYsInJlbGF5X2lkIjoiYmYzYjFiOGQtYWE2ZC00MzM5LWJiNDgtZWM5MmZiYmI5YzRjIiwicHVibGljX2tleSI6IlBRREE4YWE3WUh0UGFHekFSbDdEVFRLaHRCM2FVaGlLX0hwa2k2a0owQkkiLCJyYW5kIjoiakVZaVZOVXY5OUM5YXYtN3BGamdFM0FpYkNLY1NvU0NXQUd5bmZlUmIxZnRIQVRhbkdVNkhxTUJZWmRoRExnZm02bERYV2RHQm5JejF4WUlqOU96THcifQ:Q3IjeTLLgy2neK9Bd6aoG7t0I2XqBqHiLZ5ibhQ3Jt0Q1ST8ei_ufEYPsePzfclFaZLe631MNYESjCnfVkBf0A)
2025-07-08T12:06:26Z [relay_server::actors::upstream] DEBUG: sending register challenge response
2025-07-08T12:06:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)
2025-07-08T12:06:26Z [relay_server::actors::upstream] DEBUG: scheduling authentication retry in 60 seconds

[web]
web_1 | /usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:187: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS['github-login.extended-permissions'] instead.
web_1 | warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
web_1 | 12:02:35 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 12:02:52 [INFO] sentry.plugins.github: apps-not-configured
web_1 | *** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jul 8 12:02:53 2025] ***
web_1 | compiled with version: 8.3.0 on 29 June 2021 03:47:25
web_1 | os: Linux-5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023
web_1 | nodename: 8dd1ed5113c6
web_1 | machine: x86_64
web_1 | clock source: unix
web_1 | detected number of CPU cores: 4
web_1 | current working directory: /
web_1 | detected binary path: /usr/local/bin/uwsgi
web_1 | !!! no internal routing support, rebuild with pcre support !!!
web_1 | your memory page size is 4096 bytes
web_1 | detected max file descriptor number: 1048576
web_1 | lock engine: pthread robust mutexes
web_1 | thunder lock: enabled
web_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:9000 fd 3
web_1 | Python version: 3.6.13 (default, Jun 23 2021, 16:13:24) [GCC 8.3.0]
web_1 | Set PythonHome to /usr/local
web_1 | Python main interpreter initialized at 0x55e182ea3fd0
web_1 | python threads support enabled
web_1 | your server socket listen backlog is limited to 100 connections
web_1 | your mercy for graceful operations on workers is 60 seconds
web_1 | setting request body buffering size to 65536 bytes
web_1 | mapped 1087040 bytes (1061 KB) for 6 cores
web_1 | *** Operational MODE: preforking+threaded ***
web_1 | spawned uWSGI master process (pid: 21)
web_1 | spawned uWSGI worker 1 (pid: 25, cores: 2)
web_1 | spawned uWSGI worker 2 (pid: 26, cores: 2)
web_1 | spawned uWSGI worker 3 (pid: 27, cores: 2)
web_1 | /usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:187: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS['github-login.extended-permissions'] instead.
web_1 | warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
web_1 | /usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:187: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS['github-login.extended-permissions'] instead.
web_1 | warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
web_1 | /usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:187: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS['github-login.extended-permissions'] instead.
web_1 | warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
web_1 | 12:03:02 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 12:03:02 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 12:03:03 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 12:03:22 [INFO] sentry.plugins.github: apps-not-configured
web_1 | WSGI app 0 (mountpoint='') ready in 29 seconds on interpreter 0x55e182ea3fd0 pid: 27 (default app)
web_1 | /usr/local/lib/python3.6/site-packages/sentry/utils/strings.py:45: DeprecationWarning: invalid escape sequence '{'
web_1 | "unicode-escape", "unicode-escape-recovery"
web_1 | 12:03:23 [INFO] sentry.plugins.github: apps-not-configured
web_1 | 12:03:23 [INFO] sentry.plugins.github: apps-not-configured
web_1 | WSGI app 0 (mountpoint='') ready in 30 seconds on interpreter 0x55e182ea3fd0 pid: 25 (default app)
web_1 | WSGI app 0 (mountpoint='') ready in 30 seconds on interpreter 0x55e182ea3fd0 pid: 26 (default app)
web_1 | /usr/local/lib/python3.6/site-packages/sentry/utils/strings.py:45: DeprecationWarning: invalid escape sequence '{'
web_1 | "unicode-escape", "unicode-escape-recovery"
web_1 | /usr/local/lib/python3.6/site-packages/sentry/utils/strings.py:45: DeprecationWarning: invalid escape sequence '{'
web_1 | "unicode-escape", "unicode-escape-recovery"

[kafka]
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1 | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.28.0.4:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.28.0.11:33262, server: zookeeper/172.28.0.4:2181
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.28.0.4:2181, sessionid = 0x1001ac210580000, negotiated timeout = 40000
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1001ac210580000 closed
kafka_1 | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1001ac210580000
kafka_1 | ===> Launching ...
kafka_1 | ===> Launching kafka ...
kafka_1 | [2025-07-08 12:02:36,989] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2025-07-08 12:02:41,861] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1 | [2025-07-08 12:02:41,861] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1 | [2025-07-08 12:02:47,248] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1 | [2025-07-08 12:02:47,591] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1 | [2025-07-08 12:02:49,900] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2025-07-08 12:02:50,166] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1 | [2025-07-08 12:02:50,168] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1 | [2025-07-08 12:02:50,919] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1 | [2025-07-08 12:02:50,984] INFO Stat of the created znode at /brokers/ids/1001 is: 16913,16913,1751976170975,1751976170975,1,0,0,72087014838108161,180,0,16913
kafka_1 | (kafka.zk.KafkaZkClient)
kafka_1 | [2025-07-08 12:02:51,001] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 16913 (kafka.zk.KafkaZkClient)
kafka_1 | [2025-07-08 12:02:52,993] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2025-07-08 12:02:53,897] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)

[zookeeper]
zookeeper_1 | ===> ENV Variables ...
zookeeper_1 | ALLOW_UNSIGNED=false
zookeeper_1 | COMPONENT=zookeeper
zookeeper_1 | CONFLUENT_DEB_VERSION=1
zookeeper_1 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1 | CONFLUENT_VERSION=5.5.0
zookeeper_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1 | HOME=/root
zookeeper_1 | HOSTNAME=352e89d5e5d5
zookeeper_1 | KAFKA_OPTS=-Dzookeeper.4lw.commands.whitelist=ruok
zookeeper_1 | KAFKA_VERSION=
zookeeper_1 | LANG=C.UTF-8
zookeeper_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1 | PWD=/
zookeeper_1 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1 | PYTHON_VERSION=2.7.9-1
zookeeper_1 | SCALA_VERSION=2.12
zookeeper_1 | SHLVL=1
zookeeper_1 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1 | _=/usr/bin/env
zookeeper_1 | ===> User
zookeeper_1 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1 | ===> Configuring ...
zookeeper_1 | ===> Running preflight checks ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1 | ===> Launching ...
zookeeper_1 | ===> Launching zookeeper ...
zookeeper_1 | [2025-07-08 12:02:26,736] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | [2025-07-08 12:02:29,414] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | [2025-07-08 12:02:29,417] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)

Our Observations:

• We verified the connectivity from relay container with web and it's seems working properly but still we are seeing the caused by: error sending request for url (http://web:9000/api/0/relays/register/response/): connection error: Connection reset by peer (os error 104)

root@sentry:/opt/sentry# docker-compose exec relay bash
root@a2fe2ad35c74:/work# curl http://web:9000/api/0/relays/register/challenge/
{"detail":"Method "GET" not allowed."}root@a2fe2ad35c74:/work#
root@a2fe2ad35c74:/work#
root@a2fe2ad35c74:/work#
root@a2fe2ad35c74:/work# curl http://web:9000/api/0/relays/register/response/
{"detail":"Method "GET" not allowed."}root@a2fe2ad35c74:/work#
root@a2fe2ad35c74:/work#

@aldy505 Can you please suggest on this ? are we missing any other components to verify to resolve this ?

Expected Results:
After the upgrade, we expect to see the latest events generated by the automated scripts successfully appearing in Sentry.