Skip to content

Commit 9d33daf

Browse files
docs: Use registry with SQLAlchemy external dialect (feast-dev#3402)
Signed-off-by: Khris Richardson <[email protected]> Signed-off-by: Khris Richardson <[email protected]>
1 parent c0ca7e4 commit 9d33daf

File tree

1 file changed

+50
-1
lines changed

1 file changed

+50
-1
lines changed

docs/tutorials/using-scalable-registry.md

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,59 @@ registry:
3333
3434
Specifically, the registry_type needs to be set to sql in the registry config block. On doing so, the path should refer to the [Database URL](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls) for the database to be used, as expected by SQLAlchemy. No other additional commands are currently needed to configure this registry.
3535
36+
Should you choose to use a database technology that is compatible with one of
37+
Feast's supported registry backends, but which speaks a different dialect (e.g.
38+
`cockroachdb`, which is compatible with `postgres`) then some further
39+
intervention may be required on your part.
40+
41+
`SQLAlchemy`, used by the registry, may not be able to detect your database
42+
version without first updating your DSN scheme to the appropriate
43+
[DBAPI/dialect combination](https://docs.sqlalchemy.org/en/14/glossary.html#term-DBAPI).
44+
When this happens, your database is likely using what is referred to as an
45+
[external dialect](https://docs.sqlalchemy.org/en/14/dialects/#external-dialects)
46+
in `SQLAlchemy` terminology. See your database's documentation for examples on
47+
how to set its scheme in the Database URL.
48+
49+
`Psycopg2`, which is the database library leveraged by the online and offline
50+
stores, is not impacted by the need to speak a particular dialect, and so the
51+
following only applies to the registry.
52+
53+
If you are not running Feast in a container, to accomodate `SQLAlchemy`'s need
54+
to speak an external dialect, install additional Python modules like we do as
55+
follows using `cockroachdb` for example:
56+
57+
```shell
58+
pip install sqlalchemy-cockroachdb
59+
```
60+
61+
If you are running Feast in a container, you will need to create a custom image
62+
like we do as follows, again using `cockroachdb` as an example:
63+
64+
```shell
65+
cat <<'EOF' >Dockerfile
66+
ARG DOCKER_IO_FEASTDEV_FEATURE_SERVER
67+
FROM docker.io/feastdev/feature-server:${DOCKER_IO_FEASTDEV_FEATURE_SERVER}
68+
ARG PYPI_ORG_PROJECT_SQLALCHEMY_COCKROACHDB
69+
RUN pip install -I --no-cache-dir \
70+
sqlalchemy-cockroachdb==${PYPI_ORG_PROJECT_SQLALCHEMY_COCKROACHDB}
71+
EOF
72+
73+
export DOCKER_IO_FEASTDEV_FEATURE_SERVER=0.27.1
74+
export PYPI_ORG_PROJECT_SQLALCHEMY_COCKROACHDB=1.4.4
75+
76+
docker build \
77+
--build-arg DOCKER_IO_FEASTDEV_FEATURE_SERVER \
78+
--build-arg PYPI_ORG_PROJECT_SQLALCHEMY_COCKROACHDB \
79+
--tag ${MY_REGISTRY}/feastdev/feature-server:${DOCKER_IO_FEASTDEV_FEATURE_SERVER} .
80+
```
81+
82+
If you are running Feast in Kubernetes, set the `image.repository` and
83+
`imagePullSecrets` Helm values accordingly to utilize your custom image.
84+
3685
There are some things to note about how the SQL registry works:
3786
- Once instantiated, the Registry ensures the tables needed to store data exist, and creates them if they do not.
3887
- Upon tearing down the feast project, the registry ensures that the tables are dropped from the database.
3988
- The schema for how data is laid out in tables can be found . It is intentionally simple, storing the serialized protobuf versions of each Feast object keyed by its name.
4089

4190
## Example Usage: Concurrent materialization
42-
The SQL Registry should be used when materializing feature views concurrently to ensure correctness of data in the registry. This can be achieved by simply running feast materialize or feature_store.materialize multiple times using a correctly configured feature_store.yaml. This will make each materialization process talk to the registry database concurrently, and ensure the metadata updates are serialized.
91+
The SQL Registry should be used when materializing feature views concurrently to ensure correctness of data in the registry. This can be achieved by simply running feast materialize or feature_store.materialize multiple times using a correctly configured feature_store.yaml. This will make each materialization process talk to the registry database concurrently, and ensure the metadata updates are serialized.

0 commit comments

Comments
 (0)