Skip to content

Commit bbae1b8

Browse files
josephcummingsjosephcummings
committed
Move replicator docs to replicator repository
1 parent 12e020d commit bbae1b8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+4137
-7
lines changed

‎.github/dependabot.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: npm
4+
directory: "/docs"
5+
schedule:
6+
interval: daily
7+
time: '20:00'
8+
open-pull-requests-limit: 10
9+
- package-ecosystem: bundler
10+
directory: "/docs"
11+
schedule:
12+
interval: daily
13+
time: '20:00'
14+
open-pull-requests-limit: 10

‎.github/workflows/docs-build.yml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Docs / Build Hugo Site
2+
on:
3+
push:
4+
branches:
5+
- 'master'
6+
pull_request:
7+
paths-ignore:
8+
- "**.md"
9+
- ".github/**"
10+
- ".gitignore"
11+
- ".gitattributes"
12+
branches:
13+
- 'master'
14+
jobs:
15+
hugo:
16+
name: Build Hugo Site
17+
runs-on: ubuntu-latest
18+
defaults:
19+
run:
20+
shell: bash
21+
working-directory: ./docs
22+
steps:
23+
-
24+
name: Checkout
25+
uses: actions/checkout@v3
26+
-
27+
name: Setup Hugo
28+
uses: peaceiris/actions-hugo@v2
29+
with:
30+
hugo-version: 'latest'
31+
extended: true
32+
-
33+
name: Install Node
34+
uses: actions/setup-node@v4
35+
with:
36+
node-version: 19.9.0
37+
-
38+
name: Install Dependencies
39+
run: npm install
40+
-
41+
name: Build Hugo Site
42+
run: hugo --minify

‎.github/workflows/build-docker.yml renamed to ‎.github/workflows/replicator-build.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Build Docker Image
1+
name: Replicator / Build Docker Image
22
on:
33
push:
44
paths:

‎.github/workflows/publish-docker.yml renamed to ‎.github/workflows/replicator-publish-docker.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Publish Docker Image
1+
name: Replicator / Publish Docker Image
22
on:
33
push:
44
paths:

‎.github/workflows/helm.yml renamed to ‎.github/workflows/replicator-publish-helm.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Publish Helm Chart
1+
name: Replicator / Publish Helm Chart
22
on:
33
push:
44
paths:

‎.gitignore

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -331,4 +331,11 @@ exclusive.lck
331331
.vscode
332332
version.txt
333333
/public/
334-
**/checkpoint
334+
**/checkpoint
335+
336+
# Hugo Docs
337+
docs/public/
338+
docs/resources/
339+
docs/node_modules/
340+
docs/tech-doc-hugo
341+
docs/.DS_Store

‎README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,13 @@ EventStoreDB readers implement one additional filter, which executes during the
1616

1717
## Build
1818

19-
On a standard x64 architecture:
20-
2119
```sh
2220
docker build .
2321
```
2422

25-
On an arm64 / Mac apple silicium:
23+
The default target architecture is amd64 (x86_64).
24+
25+
You can build targeting arm64 (e.g to execute on Apple Silicon) like so:
2626

2727
```sh
2828
docker build --build-arg RUNTIME=linux-arm64 .

‎docs/.hugo_build.lock

Whitespace-only changes.

‎docs/assets/scss/_variables_project.scss

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
/*
2+
3+
Add styles or override variables from the theme here.
4+
5+
*/
6+

‎docs/content/_index.html

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
+++
2+
title = "Replicator"
3+
linkTitle = "Event Store Replicator"
4+
+++
5+
6+
{{< blocks/cover title="Event Store Replicator" image_anchor="top" height="max" color="primary" >}}
7+
<div class="mx-auto">
8+
<a class="btn btn-lg btn-primary mr-3 mb-4" href="{{< relref "/docs" >}}">
9+
Learn More <i class="fas fa-arrow-alt-circle-right ml-2"></i>
10+
</a>
11+
<a class="btn btn-lg btn-secondary mr-3 mb-4" href="{{< ref "helm" >}}">
12+
Quick Start <i class="fas fa-rocket ml-2"></i>
13+
</a>
14+
<p class="lead mt-5">Replicate your events with ease</p>
15+
</div>
16+
{{< /blocks/cover >}}
17+
18+
{{% blocks/lead color="primary" %}}
19+
Migrate from EventStoreDB on-prem to Event Store Cloud by replicating events.
20+
Keep your data safe by replicating events to a backup cluster.
21+
One-time or continuous replication, filtering events, schema and data transformations.
22+
{{% /blocks/lead %}}
23+
24+
{{< blocks/section color="dark" type="row" >}}
25+
{{% blocks/feature icon="fas fa-plug" title="Multi-protocol" %}}
26+
Read and write to EventStoreDB v5 and v20+, using TCP or gRPC protocol.
27+
Both secure and unsecure connections are supported.
28+
{{% /blocks/feature %}}
29+
30+
{{% blocks/feature icon="fas fa-filter" title="Filters" url="Filters" %}}
31+
Filter out obsolete and deleted events with scavenge, stream name, and event type filter. You can even use transformations for advanced filtering!
32+
{{% /blocks/feature %}}
33+
34+
{{% blocks/feature icon="fas fa-random" title="Transformations" url="Transforms" %}}
35+
Upgrade your event schema, remove or add fields, change values, enrich events with information from external sources.
36+
{{% /blocks/feature %}}
37+
38+
{{< /blocks/section >}}

‎docs/content/docs/Concepts/_index.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
title: "Concepts"
3+
linkTitle: "Concepts"
4+
date: 2021-03-05
5+
weight: 2
6+
description: >
7+
Find out some basic concepts of Replicator.
8+
---
9+
10+
Replicator is designed to copy events from one place to another, which sounds like a relatively simple task. Still, there are some concepts you need to understand before using the tool.
11+
12+
## Reader
13+
14+
Reader is an adapter for the infrastructure, where you want to copy events _from_. The _reader_ reads from a _source_.
15+
16+
Currently, we support readers for EventStoreDB, using TCP and gRPC protocols. Each reader type requires its own configuration, which is usually just a connection string, specific to each reader type.
17+
18+
The reader always reads events in sequence, but all the readers support batched reads.
19+
20+
There is only one reader per running Replicator instance.
21+
22+
## Sink and writers
23+
24+
Reader is an adapter for the infrastructure, where you want to copy events _to_. The _sink_ has one or more _writers_. By using multiple writers, one sink can improve performance by parallelising writes.
25+
26+
When using one writer for a sink, the order of events in the target remains exactly the same as it was in the source.
27+
28+
When using more than one writer, the global order of events in the source cannot be guaranteed. However, multiple writers also enable partitioning. The default partition key is the stream name, which guarantees the order of events in each stream.
29+
30+
You can only have one sink per running Replicator instance, but it might have multiple writers.
31+
32+
## Checkpoint
33+
34+
A running Replicator instance progresses linearly over a given stream of events, so it knows at any time, which events were already processed. As the process might be shut down for different reasons, it needs to maintain the last processed event position, so in case of restart, Replicator will start from there, and not from the very beginning. This way, you don't get duplicated events in the sink, and you can be sure that the replication process will eventually be completed.
35+
36+
The location of the last processed event in the source is known as _checkpoint_. Replicator supports storing the checkpoint in [different stores]({{< ref "checkpoints" >}}). If you want to run the replication again, from the same source, using the same Replicator instance, you need to delete the checkpoint file.
37+
38+
## Filters
39+
40+
As you might want to ignore some events during replication, Replicator supports different [filters]({{< ref "filters" >}}). Filters allow you to cover cases like preventing some obsolete events from being replicated, or splitting one source to two targets. In the latter case, you can run two Replicator instances with different filters, so events will be distributed to different sinks.
41+
42+
## Transforms
43+
44+
After being in production for a while, most systems accumulate legacy data. You might want to remove some of it using filters, but you might also want to keep the data in a different format. Typical scenarios include evolution of event schema, missing fields, incorrect data format, oversharing (sensitive unprotected information), etc.
45+
46+
These cases can be handler by using [transforms]({{< ref "transforms" >}}), which allow you to change any part of the event that comes from the source, before writing it to the sink.
47+
48+

‎docs/content/docs/Configuration/_index.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
title: "Configuration"
3+
linkTitle: "Configuration"
4+
date: 2021-03-05
5+
weight: 5
6+
description: >
7+
Replicator configuration file explained.
8+
---
9+
10+
Replicator uses a configuration file in YAML format. The file must be called `appsettings.yaml` and located in the `config` subdirectory, relative to the tool working directory.
11+
12+
The settings file has the `replicator` root level, all settings are children to that root. It allows using the same format for the values override file when using Helm.
13+
14+
Available configuration options are:
15+
16+
| Option | Description |
17+
| :----- | :---------- |
18+
| `replicator.reader.connectionString` | Connection string for the source cluster or instance |
19+
| `replicator.reader.protocol` | Reader protocol (`tcp` or `grpc`) |
20+
| `replicator.reader.pageSize` | Reader page size (only applicable for TCP protocol |
21+
| `replicator.sink.connectionString` | Connection string for the target cluster or instance |
22+
| `replicator.sink.protocol` | Writer protocol (`tcp` or `grpc`) |
23+
| `replicator.sink.partitionCount` | Number of [partitioned]({{% ref "writers" %}}) concurrent writers |
24+
| `replicator.sink.partitioner` | Custom JavaScript [partitioner]({{% ref "writers" %}}) |
25+
| `replicator.sink.bufferSize` | Size of the sink buffer, `1000` events by default |
26+
| `replicator.scavenge` | Enable real-time [scavenge]({{% ref "scavenge" %}}) |
27+
| `replicator.runContinuously` | Set to `false` if you want Replicator to stop when it reaches the end of `$all` stream. Default is `true`, so the replication continues until you stop it explicitly. |
28+
| `replicator.filters` | Add one or more of provided [filters]({{% ref "filters" %}}) |
29+
| `replicator.transform` | Configure the [event transformation]({{% ref "Transforms" %}}) |
30+
| `replicator.transform.bufferSize` | Size of the prepare buffer (filtering and transformations), `1000` events by default |
31+
32+
## Enable verbose logging
33+
34+
You can enable debug-level logging by setting the `REPLICATOR_DEBUG` environment variable to any value.
35+
36+
## Example configuration
37+
38+
The following example configuration will instruct Replicator to read all the events from a local cluster with three nodes (`es1.acme.org`, `es2.acme.org` and `es3.acme.org`) using TCP protocol, and copy them over to the Event Store Cloud cluster with cluster ID `c2etr1lo9aeu6ojco781` using gRPC protocol. Replicator will also call an HTTP transformation function at `https://my.acme.org/transform`.
39+
40+
The global order of events will be the same, as `partitionCount` is set to one.
41+
42+
Scavenge filter is disabled, so Replicator will also copy deleted events, which haven't been scavenged by the server yet.
43+
44+
```yaml
45+
replicator:
46+
reader:
47+
protocol: tcp
48+
connectionString: "GossipSeeds=es1.acme.org:2113,es2.acme.org:2113,es3.acme.org:2113; HeartBeatTimeout=500; DefaultUserCredentials=admin:changeit; UseSslConnection=false;"
49+
pageSize: 2048
50+
sink:
51+
protocol: grpc
52+
connectionString: "esdb://admin:[email protected]:2113"
53+
partitionCount: 1
54+
transform:
55+
type: http
56+
config: https://my.acme.org/transform
57+
scavenge: false
58+
filters: []
59+
checkpoint:
60+
path: "./checkpoint"
61+
```
62+

‎docs/content/docs/Deployment/Docker/_index.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
title: "Run Replicator in Docker"
3+
linkTitle: "Docker"
4+
date: 2021-03-05
5+
description: >
6+
Running Replicator with Docker Compose
7+
---
8+
9+
You can run Replicator using [Docker Compose](https://docs.docker.com/compose/), on any machine, which has Docker installed.
10+
11+
We prepared a complete set of files for this scenario. You find those files in the [Replicator repository](https://github.com/EventStore/replicator/tree/master/compose).
12+
13+
The Compose file includes the following components:
14+
- Replicator itself
15+
- Prometheus, pre-configured to scrape Replicator metrics endpoint
16+
- Grafana, pre-configured to use Prometheus, with the Replicator dashboard included
17+
18+
## Configuration
19+
20+
Before spinning up this setup, you need to change the `replicator.yml` file. Find out about Replicator settings on the [Configuration]({{< ref "Configuration" >}}) page. We included a [sample](https://github.com/EventStore/replicator/blob/02736f6e3dd18e41d5536f26ca4f9497733d5f3f/compose/replicator.yml) configuration file to the repository.
21+
22+
{{< alert >}}The sample configuration file includes a JavaScript transform configuration as an example. It is not suitable for production purposes, so make sure you remove it from your configuration.{{< /alert >}}
23+
24+
The sample configuration enables verbose logging using the `REPLICATOR_DEBUG` environment variable. For production deployments, you should remove it from the configuration.
25+
26+
## Monitoring
27+
28+
When you start all the component using `docker-compose up`, you'd be able to check the Replicator web UI by visiting [http://localhost:5000](http://localhost:5000), as well as Grafana at [http://localhost:3000](http://localhost:3000). Use `admin`/`admin` default credentials for Grafana. The Replicator dashboard is included in the deployment, so you can find it in the [dashboards list](http://localhost:3000/dashboards).
29+
30+
{{< alert >}}Watch out for the replication gap and ensure that it decreases.{{< /alert >}}
31+
32+
![Grafana dashboard](grafana.png)
33+

‎docs/content/docs/Deployment/Docker/grafana.png

448 KB
Loading

‎docs/content/docs/Deployment/Kubernetes/_index.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
title: "Kubernetes"
3+
linkTitle: "Kubernetes"
4+
date: 2021-03-05
5+
description: >
6+
Deploying Replicator to a Kubernetes cluster
7+
---
8+
9+
You can run Replicator in a Kubernetes cluster in the same cloud as your managed EventStoreDB cloud cluster. The Kubernetes cluster workloads must be able to reach the managed EventStoreDB cluster. Usually, with a proper VPC (or VN) peering between your VPC and Event Store Cloud network, it works without issues.
10+
11+
We provide guidelines about connecting managed Kubernetes clusters:
12+
- [Google Kubernetes Engine](https://developers.eventstore.com/cloud/use/kubernetes/gke.html)
13+
- [AWS EKS](https://developers.eventstore.com/cloud/use/kubernetes/eks.html)
14+
- [Azure AKS](https://developers.eventstore.com/cloud/use/kubernetes/aks.html)

0 commit comments

Comments
 (0)