|
| 1 | +# Dragonfly online store |
| 2 | + |
| 3 | +## Description |
| 4 | + |
| 5 | +[Dragonfly](https://github.com/dragonflydb/dragonfly) is a modern in-memory datastore that implements novel algorithms and data structures on top of a multi-threaded, share-nothing architecture. Thanks to its API compatibility, Dragonfly can act as a drop-in replacement for Redis. Due to Dragonfly's hardware efficiency, you can run a single node instance on a small 8GB instance or scale vertically to large 768GB machines with 64 cores. This greatly reduces infrastructure costs as well as architectural complexity. |
| 6 | + |
| 7 | +Similar to Redis, Dragonfly can be used as an online feature store for Feast. |
| 8 | + |
| 9 | +## Using Dragonfly as a drop-in Feast online store instead of Redis |
| 10 | + |
| 11 | +Make sure you have Python and `pip` installed. |
| 12 | + |
| 13 | +Install the Feast SDK and CLI |
| 14 | + |
| 15 | +`pip install feast` |
| 16 | + |
| 17 | +In order to use Dragonfly as the online store, you'll need to install the redis extra: |
| 18 | + |
| 19 | +`pip install 'feast[redis]'` |
| 20 | + |
| 21 | +### 1. Create a feature repository |
| 22 | + |
| 23 | +Bootstrap a new feature repository: |
| 24 | + |
| 25 | +``` |
| 26 | +feast init feast_dragonfly |
| 27 | +cd feast_dragonfly/feature_repo |
| 28 | +``` |
| 29 | + |
| 30 | +Update `feature_repo/feature_store.yaml` with the below contents: |
| 31 | + |
| 32 | +``` |
| 33 | +project: feast_dragonfly |
| 34 | +registry: data/registry.db |
| 35 | +provider: local |
| 36 | +online_store: |
| 37 | +type: redis |
| 38 | +connection_string: "localhost:6379" |
| 39 | +``` |
| 40 | + |
| 41 | +### 2. Start Dragonfly |
| 42 | + |
| 43 | +There are several options available to get Dragonfly up and running quickly. We will be using Docker for this tutorial. |
| 44 | + |
| 45 | +`docker run --network=host --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly` |
| 46 | + |
| 47 | +### 3. Register feature definitions and deploy your feature store |
| 48 | + |
| 49 | +`feast apply` |
| 50 | + |
| 51 | +The `apply` command scans python files in the current directory (`example_repo.py` in this case) for feature view/entity definitions, registers the objects, and deploys infrastructure. |
| 52 | +You should see the following output: |
| 53 | + |
| 54 | +``` |
| 55 | +.... |
| 56 | +Created entity driver |
| 57 | +Created feature view driver_hourly_stats_fresh |
| 58 | +Created feature view driver_hourly_stats |
| 59 | +Created on demand feature view transformed_conv_rate |
| 60 | +Created on demand feature view transformed_conv_rate_fresh |
| 61 | +Created feature service driver_activity_v1 |
| 62 | +Created feature service driver_activity_v3 |
| 63 | +Created feature service driver_activity_v2 |
| 64 | +``` |
| 65 | + |
| 66 | +## Functionality Matrix |
| 67 | + |
| 68 | +The set of functionality supported by online stores is described in detail [here](overview.md#functionality). |
| 69 | +Below is a matrix indicating which functionality is supported by the Redis online store. |
| 70 | + |
| 71 | +| | Redis | |
| 72 | +| :-------------------------------------------------------- | :---- | |
| 73 | +| write feature values to the online store | yes | |
| 74 | +| read feature values from the online store | yes | |
| 75 | +| update infrastructure (e.g. tables) in the online store | yes | |
| 76 | +| teardown infrastructure (e.g. tables) in the online store | yes | |
| 77 | +| generate a plan of infrastructure changes | no | |
| 78 | +| support for on-demand transforms | yes | |
| 79 | +| readable by Python SDK | yes | |
| 80 | +| readable by Java | yes | |
| 81 | +| readable by Go | yes | |
| 82 | +| support for entityless feature views | yes | |
| 83 | +| support for concurrent writing to the same key | yes | |
| 84 | +| support for ttl (time to live) at retrieval | yes | |
| 85 | +| support for deleting expired data | yes | |
| 86 | +| collocated by feature view | no | |
| 87 | +| collocated by feature service | no | |
| 88 | +| collocated by entity key | yes | |
| 89 | + |
| 90 | +To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix). |
0 commit comments