Skip to content
This repository was archived by the owner on Feb 4, 2021. It is now read-only.

Add retrieval method for stream subset by date #10

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

andrewhyson
Copy link

Add method to retrieve commits from a subset of streams that occur after a specified point in time.
Partner method for GetStreams - GetStreamCommitsFrom.
Note: I've added passing unit tests for the new method - however, unit tests that were previously failing (or in an indeterminable state) before this work are still in this state following this commit.

…fter a specified point in time.

Partner method for GetStreams - GetStreamCommitsFrom
Copy link

@Dan-Dan-Man Dan-Dan-Man left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks generally okay to me. Just to clarify, we're adding this new method so we can replay particular streams from a particular point in time? There is already a method there to replay all streams from a certain point in time. If we're recovering from disaster, I would have thought we'd want to replay all streams?

@andrewhyson
Copy link
Author

Hi @Dan-Dan-Man, Yes I saw the various methods for streams by date (e.g. GetFrom) - however, I was looking for a partner to your custom GetStreams method that takes an array of streamIds.

This is so we can still support the distributed nature of the replay. By the time this method is called, the replay tool has already established the IDs of streams with changed commits (via an ad-hoc query in Mnemosyne) and distributed these via the queue to the workers. It is the workers that takes this aggregateID batch and then request the commits for them (this new method). I wanted this call to also be limited by date, so that streams with large history would not slow the process down.

Does that wound right or do I have the wrong end of the stick?

@Dan-Dan-Man
Copy link

@andrewhyson Yeah that makes sense, wasn't sure what context this method was called and that you only need the streams that the worker cares about in that moment in time

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants