docs(self-hosted): explain self-hosted data flow #13745
Conversation
|
@aldy505 is attempting to deploy a commit to the Sentry Team on Vercel. A member of the Team first needs to authorize it. |
| ### Event Ingestion Pipeline | ||
|
|
||
| 1. Events from the SDK is sent to the `relay` service. | ||
| 2. Relay parses the incoming envelope, validates whether the DSN and Project ID is valid. It reads project config data from `redis`. | ||
| 3. Relay build a new payload to be consumed by Sentry ingest consumers, and send it to `kafka`. | ||
| 4. Sentry `ingest-*` consumers ( with `*` [wildcard] being the event type [errors, transaction, profiles, etc]) consumes the event, cache it in `redis` and start the `preprocess_event` task. | ||
| 5. The `preprocess_event` task symbolicate stack traces with `symbolicator` service, and process event according to the event type's logic. | ||
| 6. The `preprocess_event` task saves the event payload to nodestore (the default is to `postgres`). | ||
| 7. The `preprocess_event` task publishes the event to `kafka` to `events` topic. | ||
|
|
||
| ### Event Processing Pipeline | ||
|
|
||
| 1. The `snuba-consumer` service consumes events from `events` topic and process them. Some are being republished to `post-process-forwarder` topic in `kafka`. Others are being written to `clickhouse`. | ||
| 2. The Sentry consumer that consumes `post-process-forwarder` topic process the events, and republish it to `kafka`. |
There was a problem hiding this comment.
I'm not sure about these. Can anyone take a look? @markstory @untitaker @hubertdeng123 @BYK
|
@AbhiPrasad Hey, this PR introduce a big chunk of mermaid diagram. Mermaid does not ship with pan & zoom feature by default, can you (or someone from the frontend devs that manages the docs) help put similar thing? There is an issue on mermaid: mermaid-js/mermaid#2162 |
|
I'm going to ping @chargome who helps maintain the docs - I don't think we can prioritize this over the other docs tasks atm though. We can always just ship this and then create a GH issue as a feature request. |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
Co-authored-by: Mark Story <[email protected]>
Co-authored-by: Mark Story <[email protected]>
Co-authored-by: Mark Story <[email protected]>
| ### Event Processing Pipeline | ||
|
|
||
| 1. The `snuba-consumer` service consumes events from `events` topic and process them. After being written to clickhouse, snuba publishes error & transaction events to `post-process-forwarder`. | ||
| 2. The Sentry `post-process-forwarder` consumer consumes messages and spawns a `post_process_group` task for each processed error & issue occurance. |
There was a problem hiding this comment.
Would be nice to note why we do this? Like where do they go after?
DESCRIBE YOUR PR
Closes getsentry/self-hosted#3585
Preview here: https://develop-docs-git-fork-aldy505-docs-self-hosteddata-flow.sentry.dev/self-hosted/data-flow/
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
LEGAL BOILERPLATE
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
EXTRA RESOURCES