Skip to content

Commit cb8db84

Browse files
hao-affirmhao-affirm
authored
feat: Add mysql as online store (feast-dev#3190)
* update remove debug Signed-off-by: hao-affirm <[email protected]> * add interface Signed-off-by: hao-affirm <[email protected]> * fix lint Signed-off-by: hao-affirm <[email protected]> * fix lint Signed-off-by: hao-affirm <[email protected]> * update sql Signed-off-by: hao-affirm <[email protected]> * update ci reqs Signed-off-by: hao-affirm <[email protected]> * remove pip index Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * fix unit test issue Signed-off-by: hao-affirm <[email protected]> * fix lint issue Signed-off-by: hao-affirm <[email protected]> * add entity_key_serialization_version Signed-off-by: hao-affirm <[email protected]> * update doc Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * fix makefile typo Signed-off-by: hao-affirm <[email protected]> * move to func Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> Signed-off-by: hao-affirm <[email protected]>
1 parent a2dc0d0 commit cb8db84

File tree

16 files changed

+473
-0
lines changed

16 files changed

+473
-0
lines changed

‎Makefile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,27 @@ test-python-universal-postgres-online:
231231
not test_snowflake" \
232232
sdk/python/tests
233233

234+
test-python-universal-mysql-online:
235+
PYTHONPATH='.' \
236+
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.mysql_repo_configuration \
237+
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.mysql \
238+
FEAST_USAGE=False \
239+
IS_TEST=True \
240+
python -m pytest -n 8 --integration \
241+
-k "not test_universal_cli and \
242+
not test_go_feature_server and \
243+
not test_feature_logging and \
244+
not test_reorder_columns and \
245+
not test_logged_features_validation and \
246+
not test_lambda_materialization_consistency and \
247+
not test_offline_write and \
248+
not test_push_features_to_offline_store and \
249+
not gcs_registry and \
250+
not s3_registry and \
251+
not test_universal_types and \
252+
not test_snowflake" \
253+
sdk/python/tests
254+
234255
test-python-universal-cassandra:
235256
PYTHONPATH='.' \
236257
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@
9494
* [DynamoDB](reference/online-stores/dynamodb.md)
9595
* [PostgreSQL (contrib)](reference/online-stores/postgres.md)
9696
* [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md)
97+
* [MySQL (contrib)](reference/online-stores/mysql.md)
9798
* [Providers](reference/providers/README.md)
9899
* [Local](reference/providers/local.md)
99100
* [Google Cloud Platform](reference/providers/google-cloud-platform.md)

‎docs/reference/online-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,7 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
3434
[cassandra.md](cassandra.md)
3535
{% endcontent-ref %}
3636

37+
{% content-ref url="mysql.md" %}
38+
[mysql.md](mysql.md)
39+
{% endcontent-ref %}
40+

‎docs/reference/online-stores/mysql.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# MySQL online store (contrib)
2+
3+
## Description
4+
5+
The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.
6+
7+
* Only the latest feature values are persisted
8+
9+
10+
## Example
11+
12+
{% code title="feature_store.yaml" %}
13+
```yaml
14+
project: my_feature_repo
15+
registry: data/registry.db
16+
provider: local
17+
online_store:
18+
type: mysql
19+
host: DB_HOST
20+
port: DB_PORT
21+
database: DB_NAME
22+
user: DB_USERNAME
23+
password: DB_PASSWORD
24+
```
25+
{% endcode %}
26+
27+
The full set of configuration options is available in [MySQLOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.mysql.MySQLOnlineStoreConfig).
28+
29+
## Functionality Matrix
30+
31+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
32+
Below is a matrix indicating which functionality is supported by the Mys online store.
33+
34+
| | Mys |
35+
| :-------------------------------------------------------- | :-- |
36+
| write feature values to the online store | yes |
37+
| read feature values from the online store | yes |
38+
| update infrastructure (e.g. tables) in the online store | yes |
39+
| teardown infrastructure (e.g. tables) in the online store | yes |
40+
| generate a plan of infrastructure changes | no |
41+
| support for on-demand transforms | yes |
42+
| readable by Python SDK | yes |
43+
| readable by Java | no |
44+
| readable by Go | no |
45+
| support for entityless feature views | yes |
46+
| support for concurrent writing to the same key | no |
47+
| support for ttl (time to live) at retrieval | no |
48+
| support for deleting expired data | no |
49+
| collocated by feature view | yes |
50+
| collocated by feature service | no |
51+
| collocated by entity key | no |
52+
53+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎sdk/python/docs/source/feast.infra.online_stores.contrib.mysql_online_store.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
feast.infra.online\_stores.contrib.mysql\_online\_store package
2+
===============================================================
3+
4+
Submodules
5+
----------
6+
7+
feast.infra.online\_stores.contrib.mysql\_online\_store.mysql module
8+
--------------------------------------------------------------------
9+
10+
.. automodule:: feast.infra.online_stores.contrib.mysql_online_store.mysql
11+
:members:
12+
:undoc-members:
13+
:show-inheritance:
14+
15+
Module contents
16+
---------------
17+
18+
.. automodule:: feast.infra.online_stores.contrib.mysql_online_store
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:

‎sdk/python/docs/source/feast.infra.online_stores.contrib.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Subpackages
99

1010
feast.infra.online_stores.contrib.cassandra_online_store
1111
feast.infra.online_stores.contrib.hbase_online_store
12+
feast.infra.online_stores.contrib.mysql_online_store
1213

1314
Submodules
1415
----------
@@ -29,6 +30,14 @@ feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
2930
:undoc-members:
3031
:show-inheritance:
3132

33+
feast.infra.online\_stores.contrib.mysql\_repo\_configuration module
34+
--------------------------------------------------------------------
35+
36+
.. automodule:: feast.infra.online_stores.contrib.mysql_repo_configuration
37+
:members:
38+
:undoc-members:
39+
:show-inheritance:
40+
3241
feast.infra.online\_stores.contrib.postgres module
3342
--------------------------------------------------
3443

‎sdk/python/feast/infra/online_stores/contrib/mysql_online_store/README.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Mysql Online Store
2+
Mysql is not included in current [Feast](https://github.com/feast-dev/feast) roadmap, this project intends to add Mysql support for Online Store.
3+
We create a table <project_name>_<feature_view_name> which gets updated with data on every materialize call
4+
5+
6+
#### Create a feature repository
7+
8+
```shell
9+
feast init feature_repo
10+
cd feature_repo
11+
```
12+
13+
#### Edit `feature_store.yaml`
14+
15+
set `online_store` type to be `mysql`
16+
17+
```yaml
18+
project: feature_repo
19+
registry: data/registry.db
20+
provider: local
21+
online_store:
22+
type: mysql
23+
host: 127.0.0.1 # mysql endpoint, default to 127.0.0.1
24+
port: 3306 # mysql port, default to 3306
25+
user: test # mysql user, default to test
26+
password: test # mysql password, default to test
27+
database: feast # mysql database, default to feast
28+
```
29+
30+
#### Apply the feature definitions in `example.py`
31+
32+
```shell
33+
feast -c feature_repo apply
34+
```
35+
##### Output
36+
```
37+
Registered entity driver_id
38+
Registered feature view driver_hourly_stats_view
39+
Deploying infrastructure for driver_hourly_stats_view
40+
```
41+
42+
### Materialize Latest Data to Online Feature Store (Mysql)
43+
```
44+
$ CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
45+
$ feast -c feature_repo materialize-incremental $CURRENT_TIME
46+
```
47+
#### Output
48+
```
49+
Materializing 1 feature views from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30 into the mysql online store.
50+
51+
driver_hourly_stats_view from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30:
52+
100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 120.59it/s]
53+
```
54+
55+
### Fetch the latest features for some entity id
56+
```python
57+
from pprint import pprint
58+
from feast import FeatureStore
59+
60+
store = FeatureStore(repo_path=".")
61+
feature_vector = store.get_online_features(
62+
features=[
63+
"driver_hourly_stats:conv_rate",
64+
"driver_hourly_stats:acc_rate",
65+
"driver_hourly_stats:avg_daily_trips",
66+
],
67+
entity_rows=[
68+
{"driver_id": 1004},
69+
{"driver_id": 1005},
70+
],
71+
).to_dict()
72+
pprint(feature_vector)
73+
74+
```
75+
#### Output
76+
```
77+
{'acc_rate': [0.01390857808291912, 0.4063614010810852],
78+
'avg_daily_trips': [69, 706],
79+
'conv_rate': [0.6624961495399475, 0.7595928311347961],
80+
'driver_id': [1004, 1005]}
81+
```

‎sdk/python/feast/infra/online_stores/contrib/mysql_online_store/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)