Translog architecture guide Distributed team #126416
Conversation
Closes ES-7879
kingherc
added
>docs
|
Pinging @elastic/es-docs (Team:Docs) |
|
Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing) |
kingherc
requested review from
BrianRothermich,
JeremyDahlgren,
Tim-Brooks and
arteam
|
Dear reviewers, handled all feedback, gentle reminder for review! |
There was a problem hiding this comment.
LGTM
One thing to consider, you can declare the links outside of the text and reference them by name inside the text so that they're not so disruptive when reading the MD in plain-text form.
See example here.
|
Thanks @JeremyDahlgren @nicktindall !
Unfortunately does not work if I'd like to make the linked text a code style as well (with the backticks). It only works for regular styled text. :( |
You might be able to do (if you haven't tried already, I used this elsewhere to have text other than the link ID inline) Still a bit clunky but better than the full link inline IMO |
There was a problem hiding this comment.
Still a bit clunky but better than the full link inline IMO
Thanks @nicktindall that works! Replaced all links as such.
| A [`MultiSnapshot`] can be used to iterate operations over multiple [`TranslogSnapshot`]s. | ||
| A [`TranslogWriter`] can be used to write operations to the translog. | ||
|
|
||
| #### Real-time GETs from the translog |
There was a problem hiding this comment.
I know the Recovery section isn't written yet, but may be worth at least linking to it here and saying something about how we replay the operations from the translog during recovery by just reading them sequentially.
There was a problem hiding this comment.
Good point, added in the introduction:
, so they can be replayed by just reading them sequentially from the translog during recovery in the event of ephemeral failures such as a crash or power loss.
| Each translog is a sequence of files, each identified by a translog generation ID, each containing a sequence of operations, with the last file open for writes. | ||
| The last file has a part which has been fsync'ed to disk, and a part which has been written but not necessarily fsync'ed yet to disk. | ||
| Each operation is identified by a sequence number (`seqno`), which is monotonically increased by the engine's ingestion functionality. | ||
| A [`Checkpoint`] file is also maintained, that contains, among other information, the current translog generation ID, and its last fsync'ed operation and location, the minimum translog generation ID, and the minimum and maximum sequence number of operations the sequence of translog generations include. |
There was a problem hiding this comment.
Could you explain here why the separate Checkpoint is necessary?
| Each translog is a sequence of files, each identified by a translog generation ID, each containing a sequence of operations, with the last file open for writes. | ||
| The last file has a part which has been fsync'ed to disk, and a part which has been written but not necessarily fsync'ed yet to disk. | ||
| Each operation is identified by a sequence number (`seqno`), which is monotonically increased by the engine's ingestion functionality. | ||
| A [`Checkpoint`] file is also maintained, that contains, among other information, the current translog generation ID, and its last fsync'ed operation and location, the minimum translog generation ID, and the minimum and maximum sequence number of operations the sequence of translog generations include. |
| A [`MultiSnapshot`] can be used to iterate operations over multiple [`TranslogSnapshot`]s. | ||
| A [`TranslogWriter`] can be used to write operations to the translog. | ||
|
|
||
| #### Real-time GETs from the translog |
There was a problem hiding this comment.
Good point, added in the introduction:
, so they can be replayed by just reading them sequentially from the translog during recovery in the event of ephemeral failures such as a crash or power loss.
| The last file has a part which has been fsync'ed to disk, and a part which has been written but not necessarily fsync'ed yet to disk. | ||
| Each operation is identified by a sequence number (`seqno`), which is monotonically increased by the engine's ingestion functionality. | ||
| Typically the entries in the translog are in increasing order of their sequence number, but not necessarily. | ||
| A [`Checkpoint`] file is also maintained, which is written on each fsync operation of the translog, and records important metadata and statistics about the translog, such as the current translog generation ID, its last fsync'ed operation and location, the minimum translog generation ID, and the minimum and maximum sequence number of operations the sequence of translog generations include, all of which are useful to identify the translog operations needed to be replayed upon recovery. |
There was a problem hiding this comment.
It's more than just "useful", this is essential for correctness. The Checkpoint records the last fsynced location in the translog file, i.e. it is safe to read up to the location in the checkpoint but beyond that point are only dragons.
Closes ES-7879