Explain data flow of self-hosted Sentry's architecture

[aldy505]

Problem Statement

From @kanadaj on Discord:

We need a step by step description of what data goes where and last time I checked we kinda lacked that. It's hard to know which container you have to debug if something isn't ingesting, since we have dozens of Kafka queues and processing services, and I know some of the services fees data back from Kafka into Kafka.

And yes, there is a rough outline with a chart somewhere, but it's not really specific enough at this point

Solution Brainstorm

No response

[aldy505]

Question: What do you think would best to explain this, since a diagram would probably result in duplicated info with https://develop.sentry.dev/application-architecture/overview/

Probably it's better to explain what each container does, instead of the event flow?

....or probably... just update this frequently? https://github.com/getsentry/event-ingestion-graph -- and make it more detailed, since I don't think Relay only publish to "ingest-events" topic.

[BYK]

@hubertdeng123, thoughts? (or just tag other people? Maybe @untitaker or @markstory)

[markstory]

I agree we could do a better job on documenting how the various consumers and tasks interact for ingestion. The event-ingestion-graph diagram/document might be a good place to add the additional context of which topics and consumer pods are involved.

[hubertdeng123]

IMO, if we want to explain the data flow here of Sentry's architecture in a diagram we should do so in terms of groups of containers, otherwise the diagram could get pretty overwhelming. Ingest consumers, snuba consumers, post-process-forwarders, etc. Like Mark mentioned above, mapping of topic to consumers would be really useful context to add

[aldy505]

The event-ingestion-graph diagram/document might be a good place to add the additional context of which topics and consumer pods are involved.

@markstory Would the better idea is to move the event-ingestion-graph into the sentry-docs (dev section) instead? That way, it'll get noticed relatively quick by the employee if they know some parts of the ingestion pipeline is changing.

IMO, if we want to explain the data flow here of Sentry's architecture in a diagram we should do so in terms of groups of containers, otherwise the diagram could get pretty overwhelming. Ingest consumers, snuba consumers, post-process-forwarders, etc.

@hubertdeng123 I agree with this one.