docs: Re-organizing docs slightly and adding more code examples (feast-dev#3019)

* docs: Re-organizing docs slightly and adding more code examples in overview + feature retrieval + feature view concepts

Signed-off-by: Danny Chiao <[email protected]>

* fix broken links

Signed-off-by: Danny Chiao <[email protected]>

* fix missing table info

Signed-off-by: Danny Chiao <[email protected]>

* fix empty spaces

Signed-off-by: Danny Chiao <[email protected]>

* fix empty spaces

Signed-off-by: Danny Chiao <[email protected]>

* fix nits

Signed-off-by: Danny Chiao <[email protected]>

* remove broken faq

Signed-off-by: Danny Chiao <[email protected]>

Author: adchia

docs/README.md

@@ -65,7 +65,7 @@ Explore the following resources to get started with Feast:
 * [Quickstart](getting-started/quickstart.md) is the fastest way to get started with Feast
 * [Concepts](getting-started/concepts/) describes all important Feast API concepts
 * [Architecture](getting-started/architecture-and-components/) describes Feast's overall architecture.
-* [Tutorials](tutorials/tutorials-overview.md) shows full examples of using Feast in machine learning applications.
+* [Tutorials](tutorials/tutorials-overview/) shows full examples of using Feast in machine learning applications.
 * [Running Feast with Snowflake/GCP/AWS](how-to-guides/feast-snowflake-gcp-aws/) provides a more in-depth guide to using Feast.
 * [Reference](reference/feast-cli-commands.md) contains detailed API and design documents.
 * [Contributing](project/contributing.md) contains resources for anyone who wants to contribute to Feast.

docs/SUMMARY.md

@@ -10,7 +10,7 @@
 * [Quickstart](getting-started/quickstart.md)
 * [Concepts](getting-started/concepts/README.md)
   * [Overview](getting-started/concepts/overview.md)
-  * [Data source](getting-started/concepts/data-source.md)
+  * [Data ingestion](getting-started/concepts/data-ingestion.md)
   * [Entity](getting-started/concepts/entity.md)
   * [Feature view](getting-started/concepts/feature-view.md)
   * [Feature retrieval](getting-started/concepts/feature-retrieval.md)
@@ -31,11 +31,11 @@
 
 ## Tutorials
 
-* [Overview](tutorials/tutorials-overview.md)
-* [Driver ranking](tutorials/driver-ranking-with-feast.md)
-* [Fraud detection on GCP](tutorials/fraud-detection.md)
-* [Real-time credit scoring on AWS](tutorials/real-time-credit-scoring-on-aws.md)
-* [Driver stats on Snowflake](tutorials/driver-stats-on-snowflake.md)
+* [Sample use-case tutorials](tutorials/tutorials-overview/README.md)
+  * [Driver ranking](tutorials/tutorials-overview/driver-ranking-with-feast.md)
+  * [Fraud detection on GCP](tutorials/tutorials-overview/fraud-detection.md)
+  * [Real-time credit scoring on AWS](tutorials/tutorials-overview/real-time-credit-scoring-on-aws.md)
+  * [Driver stats on Snowflake](tutorials/tutorials-overview/driver-stats-on-snowflake.md)
 * [Validating historical features with Great Expectations](tutorials/validating-historical-features.md)
 * [Using Scalable Registry](tutorials/using-scalable-registry.md)
 * [Building streaming features](tutorials/building-streaming-features.md)
@@ -50,12 +50,12 @@
   * [Load data into the online store](how-to-guides/feast-snowflake-gcp-aws/load-data-into-the-online-store.md)
   * [Read features from the online store](how-to-guides/feast-snowflake-gcp-aws/read-features-from-the-online-store.md)
 * [Running Feast in production](how-to-guides/running-feast-in-production.md)
-* [Upgrading from Feast 0.9](https://docs.google.com/document/u/1/d/1AOsr\_baczuARjCpmZgVd8mCqTF4AZ49OEyU4Cn-uTT0/edit)
 * [Upgrading for Feast 0.20+](how-to-guides/automated-feast-upgrade.md)
-* [Adding a customer provider](how-to-guides/creating-a-custom-provider.md)
-* [Adding a custom batch materialization engine](how-to-guides/creating-a-custom-materialization-engine.md)
-* [Adding a new online store](how-to-guides/adding-support-for-a-new-online-store.md)
-* [Adding a new offline store](how-to-guides/adding-a-new-offline-store.md)
+* [Customizing Feast](how-to-guides/customizing-feast/README.md)
+  * [Adding a custom batch materialization engine](how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md)
+  * [Adding a new offline store](how-to-guides/customizing-feast/adding-a-new-offline-store.md)
+  * [Adding a new online store](how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md)
+  * [Adding a custom provider](how-to-guides/customizing-feast/creating-a-custom-provider.md)
 * [Adding or reusing tests](how-to-guides/adding-or-reusing-tests.md)
 
 ## Reference

docs/getting-started/architecture-and-components/batch-materialization-engine.md

@@ -4,7 +4,6 @@ A batch materialization engine is a component of Feast that's responsible for mo
 
 A materialization engine abstracts over specific technologies or frameworks that are used to materialize data. It allows users to use a pure local serialized approach (which is the default LocalMaterializationEngine), or delegates the materialization to seperate components (e.g. AWS Lambda, as implemented by the the LambdaMaterializaionEngine).
 
-If the built-in engines are not sufficient, you can create your own custom materialization engine. Please see [this guide](../../how-to-guides/creating-a-custom-materialization-engine.md) for more details.
+If the built-in engines are not sufficient, you can create your own custom materialization engine. Please see [this guide](../../how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md) for more details.
 
 Please see [feature\_store.yaml](../../reference/feature-repository/feature-store-yaml.md#overview) for configuring engines.
-

docs/getting-started/architecture-and-components/offline-store.md

@@ -1,6 +1,6 @@
 # Offline store
 
-An offline store is an interface for working with historical time-series feature values that are stored in [data sources](../../getting-started/concepts/data-source.md).
+An offline store is an interface for working with historical time-series feature values that are stored in [data sources](../../getting-started/concepts/data-ingestion.md).
 The `OfflineStore` interface has several different implementations, such as `BigQueryOfflineStore`, each of which is backed by a different storage and compute engine.
 For more details on which offline stores are supported, please see [Offline Stores](../../reference/offline-stores/).
 

docs/getting-started/architecture-and-components/provider.md

@@ -1,10 +1,9 @@
 # Provider
 
-A provider is an implementation of a feature store using specific feature store components \(e.g. offline store, online store\) targeting a specific environment \(e.g. GCP stack\). 
+A provider is an implementation of a feature store using specific feature store components (e.g. offline store, online store) targeting a specific environment (e.g. GCP stack).
 
-Providers orchestrate various components \(offline store, online store, infrastructure, compute\) inside an environment. For example, the `gcp` provider supports [BigQuery](https://cloud.google.com/bigquery) as an offline store and [Datastore](https://cloud.google.com/datastore) as an online store, ensuring that these components can work together seamlessly. Feast has three built-in providers \(`local`, `gcp`, and `aws`\) with default configurations that make it easy for users to start a feature store in a specific environment. These default configurations can be overridden easily. For instance, you can use the `gcp` provider but use Redis as the online store instead of Datastore.
+Providers orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the `gcp` provider supports [BigQuery](https://cloud.google.com/bigquery) as an offline store and [Datastore](https://cloud.google.com/datastore) as an online store, ensuring that these components can work together seamlessly. Feast has three built-in providers (`local`, `gcp`, and `aws`) with default configurations that make it easy for users to start a feature store in a specific environment. These default configurations can be overridden easily. For instance, you can use the `gcp` provider but use Redis as the online store instead of Datastore.
 
-If the built-in providers are not sufficient, you can create your own custom provider. Please see [this guide](../../how-to-guides/creating-a-custom-provider.md) for more details.
+If the built-in providers are not sufficient, you can create your own custom provider. Please see [this guide](../../how-to-guides/customizing-feast/creating-a-custom-provider.md) for more details.
 
 Please see [feature\_store.yaml](../../reference/feature-repository/feature-store-yaml.md#overview) for configuring providers.
-

docs/getting-started/concepts/README.md

@@ -4,8 +4,8 @@
 [overview.md](overview.md)
 {% endcontent-ref %}
 
-{% content-ref url="data-source.md" %}
-[data-source.md](data-source.md)
+{% content-ref url="data-ingestion.md" %}
+[data-ingestion.md](data-ingestion.md)
 {% endcontent-ref %}
 
 {% content-ref url="entity.md" %}

docs/getting-started/concepts/data-ingestion.md

@@ -0,0 +1,86 @@
+# Data ingestion
+
+### Data source
+
+The data source refers to raw underlying data (e.g. a table in BigQuery).
+
+Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or when materializing features into an online store.
+
+Below is an example data source with a single entity (`driver`) and two features (`trips_today`, and `rating`).
+
+![Ride-hailing data source](<../../.gitbook/assets/image (16).png>)
+
+Feast supports primarily **time-stamped** tabular data as data sources. There are many kinds of possible data sources:
+
+* **Batch data sources:** ideally, these live in data warehouses (BigQuery, Snowflake, Redshift), but can be in data lakes (S3, GCS, etc). Feast supports ingesting and querying data across both.
+* **Stream data sources**: Feast does **not** have native streaming integrations. It does however facilitate making streaming features available in different environments. There are two kinds of sources:
+  * **Push sources** allow users to push features into Feast, and make it available for training / batch scoring ("offline"), for realtime feature serving ("online") or both.
+  * **\[Alpha] Stream sources** allow users to register metadata from Kafka or Kinesis sources. The onus is on the user to ingest from these sources, though Feast provides some limited helper methods to ingest directly from Kafka / Kinesis topics.
+* **(Experimental) Request data sources:** This is data that is only available at request time (e.g. from a user action that needs an immediate model prediction response). This is primarily relevant as an input into **on-demand feature views**, which allow light-weight feature engineering and combining features across sources.
+
+### Batch data ingestion
+
+Ingesting from batch sources is only necessary to power real-time models. This is done through **materialization**. Under the hood, Feast manages an _offline store_ (to scalably generate training data from batch sources) and an _online store_ (to provide low-latency access to features for real-time models).
+
+A key command to use in Feast is the `materialize_incremental` command, which fetches the latest values for all entities in the batch source and ingests these values into the online store.
+
+Materialization can be called programmatically or through the CLI:
+
+<details>
+
+<summary>Code example: programmatic scheduled materialization</summary>
+
+This snippet creates a feature store object which points to the registry (which knows of all defined features) and the online store (DynamoDB in this case), and
+
+```python
+# Define Python callable
+def materialize():
+  repo_config = RepoConfig(
+    registry=RegistryConfig(path="s3://[YOUR BUCKET]/registry.pb"),
+    project="feast_demo_aws",
+    provider="aws",
+    offline_store="file",
+    online_store=DynamoDBOnlineStoreConfig(region="us-west-2")
+  )
+  store = FeatureStore(config=repo_config)
+  store.materialize_incremental(datetime.datetime.now())
+
+# (In production) Use Airflow PythonOperator
+materialize_python = PythonOperator(
+    task_id='materialize_python',
+    python_callable=materialize,
+)
+```
+
+</details>
+
+<details>
+
+<summary>Code example: CLI based materialization</summary>
+
+#### How to run this in the CLI
+
+```bash
+CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S") 
+feast materialize-incremental $CURRENT_TIME
+```
+
+#### How to run this on Airflow
+
+```python
+# Use BashOperator
+materialize_bash = BashOperator(
+    task_id='materialize',
+    bash_command=f'feast materialize-incremental {datetime.datetime.now().replace(microsecond=0).isoformat()}',
+)
+```
+
+</details>
+
+### Stream data ingestion
+
+Ingesting from stream sources happens either via a Push API or via a contrib processor that leverages an existing Spark context.
+
+* To **push data into the offline or online stores**: see [push sources](../../reference/data-sources/push.md) for details.
+* (experimental) To **use a contrib Spark processor** to ingest from a topic, see [Tutorial: Building streaming features](../../tutorials/building-streaming-features.md)
+

docs/getting-started/concepts/data-source.md

@@ -33,14 +33,11 @@ At _training time_, users control what entities they want to look up, for exampl
 At _serving time_, users specify _entity key(s)_ to fetch the latest feature values for to power a real-time model prediction (e.g. a fraud detection model that needs to fetch the transaction user's features).
 
 {% hint style="info" %}
-**Q: Can I retrieve features for **_**all**_** entities in Feast?**
-
-Kind of. \
-
-
-In practice, this is most relevant for _batch scoring models_ (e.g. predict user churn for all existing users) that are offline only. For these use cases, Feast supports generating features for a SQL backed list of entities. There is an [open GitHub issue](https://github.com/feast-dev/feast/issues/1611) that welcomes contribution to make this a more intuitive API. 
+**Q: Can I retrieve features for all entities?**
 
+Kind of.
 
+In practice, this is most relevant for _batch scoring models_ (e.g. predict user churn for all existing users) that are offline only. For these use cases, Feast supports generating features for a SQL-backed list of entities. There is an [open GitHub issue](https://github.com/feast-dev/feast/issues/1611) that welcomes contribution to make this a more intuitive API.
 
 For _real-time feature retrieval_, there is no out of the box support for this because it would promote expensive and slow scan operations. Users can still pass in a large list of entities for retrieval, but this does not scale well.
 {% endhint %}