GitBook: [master] 80 pages modified

Author: woop

docs/SUMMARY.md

@@ -15,15 +15,33 @@
 
 ## Concepts
 
-* [Data model and concepts](concepts/data-model-and-concepts.md)
-* [Architecture and components](concepts/architecture-and-components.md)
+* [Overview](concepts/overview.md)
+* [Feature view](concepts/feature-view.md)
+* [Data model](concepts/data-model-and-concepts.md)
+* [Online Store](concepts/online-store.md)
+* [Offline Store](concepts/offline-store.md)
+* [Provider](concepts/provider.md)
+* [Architecture](concepts/architecture-and-components.md)
 
 ## Reference
 
-* [Feature repository](reference/feature-repository.md)
-* [feature\_store.yaml](reference/feature-store-yaml.md)
+* [Data Sources](reference/data-sources/README.md)
+  * [BigQuery](reference/data-sources/bigquery.md)
+  * [File](reference/data-sources/file.md)
+* [Online stores](reference/online-stores/README.md)
+  * [SQLite](reference/online-stores/sqlite.md)
+  * [Redis](reference/online-stores/redis.md)
+  * [Datastore](reference/online-stores/datastore.md)
+* [Offline stores](reference/offline-stores/README.md)
+  * [File](reference/offline-stores/file.md)
+  * [BigQuery](reference/offline-stores/untitled.md)
+* [Providers](reference/providers/README.md)
+  * [Local](reference/providers/local.md)
+  * [Google Cloud Platform](reference/providers/google-cloud-platform.md)
 * [Feast CLI reference](reference/feast-cli-commands.md)
-* [.feastignore](reference/feast-ignore.md)
+* [Feature repository](reference/feature-repository/README.md)
+  * [feature\_store.yaml](reference/feature-repository/feature-store-yaml.md)
+  * [.feastignore](reference/feature-repository/feast-ignore.md)
 * [Python API reference](http://rtd.feast.dev/)
 * [Telemetry](reference/telemetry.md)
 

docs/concepts/architecture-and-components.md

@@ -1,8 +1,8 @@
-# Architecture and components
+# Architecture
 
-![Feast 0.10 Architecture Diagram](../.gitbook/assets/image%20%284%29.png)
+![Feast Architecture Diagram](../.gitbook/assets/image%20%284%29.png)
 
-### Functionality
+#### Functionality
 
 * **Create Batch Features:** ELT/ETL systems like Spark and SQL are used to transform data in the batch store.
 * **Feast Apply:**  The user \(or CI\) publishes versioned controlled feature definitions using `feast apply`. This CLI command updates infrastructure and persists definitions in the object store registry.
@@ -13,7 +13,7 @@
 * **Prediction:** A backend system makes a request for a prediction from the model serving service.
 * **Get Online Features:** The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.
 
-### Components
+#### Components
 
 A complete Feast deployment contains the following components:
 

docs/concepts/data-model-and-concepts.md

@@ -1,87 +1,4 @@
-# Data model and concepts
-
-### Concepts
-
-The top-level namespace within Feast is a [project](data-model-and-concepts.md#project). Users define one or more [feature views](data-model-and-concepts.md#feature-view) within a project. Each feature view contains one or more [features](data-model-and-concepts.md#feature) that relate to a specific [entity](data-model-and-concepts.md#entity). A feature view must always have a [data source](data-model-and-concepts.md#data-source). This source is used during the generation of training [datasets](data-model-and-concepts.md#dataset) and when materializing feature values into the online store. 
-
-![](../.gitbook/assets/image%20%287%29.png)
-
-### Project
-
-Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment \(`dev`, `staging`, `prod`\).
-
-{% hint style="info" %}
-Projects are currently being supported for backward compatibility reasons. The concept and functionality provided by Projects may change in the future as we simplify the Feast API.
-{% endhint %}
-
-### Data Source
-
-Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or when materializing features into an online store.
-
-Below is an example data source with a single entity \(`driver`\) and two features \(`trips_today`, and `rating`\).
-
-![Ride-hailing data source](../.gitbook/assets/image%20%2816%29.png)
-
-### Entity
-
-An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.
-
-```python
-driver = Entity(name='driver', value_type=ValueType.STRING, join_key='driver_id')
-```
-
-Entities are defined as part of feature views. Entities are used to identify the primary key on which feature values should be stored and retrieved. These keys are used during the lookup of feature values from the online store and the join process in point-in-time joins. It is possible to define composite entities \(more than one entity object\) in a feature view.
-
-Entities should be reused across feature views.
-
-### Feature
-
-A feature is an individual measurable property observed on an entity. For example, a feature of a `customer` entity could be the number of transactions they have made on an average month. 
-
-Features are defined as part of feature views. Since Feast does not transform data, a feature is essentially a schema that only contains a name and a type:
-
-```python
-trips_today = Feature(
-    name="trips_today",
-    dtype=ValueType.FLOAT
-)
-```
-
-Together with [data sources](data-model-and-concepts.md#data-source), they indicate to Feast where to find your feature values, e.g., in a specific parquet file or BigQuery table. Feature definitions are also used when reading features from the feature store, using [feature references](data-model-and-concepts.md#feature-references).
-
-Feature names must be unique within a [feature view](data-model-and-concepts.md#feature-view).
-
-### Feature View
-
-A feature view is an object that represents a logical group of time-series feature data as it is found in a data source. Feature views consist of one or more entities, features, and a data source. Feature views allow Feast to model your existing feature data in a consistent way in both an offline \(training\) and online \(serving\) environment.
-
-{% tabs %}
-{% tab title="driver\_trips\_feature\_view.py" %}
-```python
-driver_stats_fv = FeatureView(
-    name="driver_activity",
-    entities=["driver"],
-    features=[
-        Feature(name="trips_today", dtype=ValueType.INT64),
-        Feature(name="rating", dtype=ValueType.FLOAT),
-    ],
-    input=BigQuerySource(
-        table_ref="feast-oss.demo_data.driver_activity"
-    )
-)
-```
-{% endtab %}
-{% endtabs %}
-
-Feature views are used during
-
-* The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.
-* Loading of feature values into an online store. Feature views determine the storage schema in the online store.
-* Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.
-
-{% hint style="info" %}
-Feast does not generate feature values. It acts as the ingestion and serving system. The data sources described within feature views should reference feature values in their already computed form.
-{% endhint %}
+# Data model
 
 ### Dataset
 
@@ -147,42 +64,3 @@ Example of an entity dataframe with feature values joined to it:
 
 ![](../.gitbook/assets/image%20%2817%29.png)
 
-### **Online Store** 
-
-The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the `materialize` command.
-
-The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.
-
-Example batch data source
-
-![](../.gitbook/assets/image%20%286%29.png)
-
-Once the above data source is materialized into Feast \(using `feast materialize`\), the feature values will be stored as follows:
-
-![](../.gitbook/assets/image%20%285%29.png)
-
-### Offline Store
-
-An offline store is a storage and compute system where historic feature data can be stored or accessed for building training datasets or for sourcing data for materialization into the online store.
-
-Offline stores are used primarily for two reasons
-
-1. Building training datasets
-2. Querying data sources for feature data in order to load these features into your online store
-
-Feast does not actively manage your offline store. Instead, you are asked to select an offline store \(like `BigQuery` or the `File` offline store\) and then to introduce batch sources from these stores using [data sources](data-model-and-concepts.md#data-source) inside feature views.
-
-Feast will use your offline store to query these sources. It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a `File` offline store, nor is it possible for a `BigQuery` offline store to query files in your local file system.
-
-Please see [feature\_store.yaml](../reference/feature-store-yaml.md#overview) for configuring your offline store.
-
-### **Provider**
-
-A provider is an implementation of a feature store using specific feature store components targeting a specific environment**.** More specifically, a provider is the target environment to which you have configured your feature store to deploy and run.
-
-Providers are built to orchestrate various components \(offline store, online store, infrastructure, compute\) inside an environment. For example, the `gcp` provider may only support `BigQuery` as an offline store and `datastore` as the online store, but it ensures that these components can work together seamlessly.
-
-Providers also come with default configurations which makes it easier for users to start a feature store in a specific environment.
-
-Please see [feature\_store.yaml](../reference/feature-store-yaml.md#overview) for configuring a provider.
-

docs/concepts/feature-view.md

@@ -0,0 +1,71 @@
+# Feature view
+
+### Feature View
+
+A feature view is an object that represents a logical group of time-series feature data as it is found in a [data source](feature-view.md#data-source). Feature views consist of one or more [entities](feature-view.md#entity), [features](feature-view.md#feature), and a [data source](feature-view.md#data-source). Feature views allow Feast to model your existing feature data in a consistent way in both an offline \(training\) and online \(serving\) environment.
+
+{% tabs %}
+{% tab title="driver\_trips\_feature\_view.py" %}
+```python
+driver_stats_fv = FeatureView(
+    name="driver_activity",
+    entities=["driver"],
+    features=[
+        Feature(name="trips_today", dtype=ValueType.INT64),
+        Feature(name="rating", dtype=ValueType.FLOAT),
+    ],
+    input=BigQuerySource(
+        table_ref="feast-oss.demo_data.driver_activity"
+    )
+)
+```
+{% endtab %}
+{% endtabs %}
+
+Feature views are used during
+
+* The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.
+* Loading of feature values into an online store. Feature views determine the storage schema in the online store.
+* Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.
+
+{% hint style="info" %}
+Feast does not generate feature values. It acts as the ingestion and serving system. The data sources described within feature views should reference feature values in their already computed form.
+{% endhint %}
+
+### Data Source
+
+Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or when materializing features into an online store.
+
+Below is an example data source with a single entity \(`driver`\) and two features \(`trips_today`, and `rating`\).
+
+![Ride-hailing data source](../.gitbook/assets/image%20%2816%29.png)
+
+### Entity
+
+An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.
+
+```python
+driver = Entity(name='driver', value_type=ValueType.STRING, join_key='driver_id')
+```
+
+Entities are defined as part of feature views. Entities are used to identify the primary key on which feature values should be stored and retrieved. These keys are used during the lookup of feature values from the online store and the join process in point-in-time joins. It is possible to define composite entities \(more than one entity object\) in a feature view.
+
+Entities should be reused across feature views.
+
+### Feature
+
+A feature is an individual measurable property observed on an entity. For example, a feature of a `customer` entity could be the number of transactions they have made on an average month. 
+
+Features are defined as part of feature views. Since Feast does not transform data, a feature is essentially a schema that only contains a name and a type:
+
+```python
+trips_today = Feature(
+    name="trips_today",
+    dtype=ValueType.FLOAT
+)
+```
+
+Together with [data sources](data-model-and-concepts.md#data-source), they indicate to Feast where to find your feature values, e.g., in a specific parquet file or BigQuery table. Feature definitions are also used when reading features from the feature store, using [feature references](data-model-and-concepts.md#feature-references).
+
+Feature names must be unique within a [feature view](data-model-and-concepts.md#feature-view).
+

docs/concepts/offline-store.md

@@ -0,0 +1,15 @@
+# Offline Store
+
+An offline store is a storage and compute system where historic feature data can be stored or accessed for building training datasets or for sourcing data for materialization into the online store.
+
+Offline stores are used primarily for two reasons
+
+1. Building training datasets
+2. Querying data sources for feature data in order to load these features into your online store
+
+Feast does not actively manage your offline store. Instead, you are asked to select an offline store \(like `BigQuery` or the `File` offline store\) and then to introduce batch sources from these stores using [data sources](data-model-and-concepts.md#data-source) inside feature views.
+
+Feast will use your offline store to query these sources. It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a `File` offline store, nor is it possible for a `BigQuery` offline store to query files in your local file system.
+
+Please see [feature\_store.yaml](../reference/feature-repository/feature-store-yaml.md#overview) for configuring your offline store.
+

docs/concepts/online-store.md

@@ -0,0 +1,16 @@
+# Online Store
+
+The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the `materialize` command.
+
+The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.
+
+Example batch data source
+
+![](../.gitbook/assets/image%20%286%29.png)
+
+Once the above data source is materialized into Feast \(using `feast materialize`\), the feature values will be stored as follows:
+
+![](../.gitbook/assets/image%20%285%29.png)
+
+### 
+

docs/concepts/overview.md

@@ -1,21 +1,16 @@
 # Overview
 
-### Concepts
+The top-level namespace within Feast is a [project](data-model-and-concepts.md#project). Users define one or more [feature views](data-model-and-concepts.md#feature-view) within a project. Each feature view contains one or more [features](data-model-and-concepts.md#feature) that relate to a specific [entity](data-model-and-concepts.md#entity). A feature view must always have a [data source](data-model-and-concepts.md#data-source), which in turn is used during the generation of training [datasets](data-model-and-concepts.md#dataset) and when materializing feature values into the online store. 
 
-[Entities](entities.md) are objects in an organization like customers, transactions, and drivers, products, etc.
+![](../.gitbook/assets/image%20%287%29.png)
 
-[Sources](sources.md) are external sources of data where feature data can be found.
+### Project
 
-[Feature Tables](feature-tables.md) are objects that define logical groupings of features, data sources, and other related metadata.
+Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment \(`dev`, `staging`, `prod`\).
 
-### Concept Hierarchy
+{% hint style="info" %}
+Projects are currently being supported for backward compatibility reasons. Projects may change in the future as we simplify the Feast API.
+{% endhint %}
 
-![](../.gitbook/assets/image%20%284%29%20%282%29%20%282%29%20%282%29%20%282%29%20%282%29%20%282%29%20%282%29%20%281%29.png)
-
-Feast contains the following core concepts:
-
-* **Projects:** Serve as a top level namespace for all Feast resources. Each project is a completely independent environment in Feast. Users can only work in a single project at a time.
-* **Entities:** Entities are the objects in an organization on which features occur. They map to your business domain \(users, products, transactions, locations\).
-* **Feature Tables:** Defines a group of features that occur on a specific entity.
-* **Features:** Individual feature within a feature table.
+### 
 

docs/concepts/provider.md

@@ -0,0 +1,10 @@
+# Provider
+
+A provider is an implementation of a feature store using specific feature store components targeting a specific environment**.** More specifically, a provider is the target environment to which you have configured your feature store to deploy and run.
+
+Providers are built to orchestrate various components \(offline store, online store, infrastructure, compute\) inside an environment. For example, the `gcp` provider supports [BigQuery](https://cloud.google.com/bigquery) as an offline store and [Datastore](https://cloud.google.com/datastore) as an online store, ensuring that these components can work together seamlessly.
+
+Providers also come with default configurations which makes it easier for users to start a feature store in a specific environment.
+
+Please see [feature\_store.yaml](../reference/feature-repository/feature-store-yaml.md#overview) for configuring providers.
+