GitBook: [master] 83 pages and 3 assets modified

Author: woop

docs/.gitbook/assets/datastore_online_example.png

@@ -30,7 +30,7 @@
   * [File](reference/data-sources/file.md)
 * [Offline stores](reference/offline-stores/README.md)
   * [File](reference/offline-stores/file.md)
-  * [BigQuery](reference/offline-stores/untitled.md)
+  * [BigQuery](reference/offline-stores/bigquery.md)
 * [Online stores](reference/online-stores/README.md)
   * [SQLite](reference/online-stores/sqlite.md)
   * [Redis](reference/online-stores/redis.md)
@@ -45,61 +45,62 @@
 * [Python API reference](http://rtd.feast.dev/)
 * [Usage](reference/usage.md)
 
-## Feast on Kubernetes
-
-* [Getting started](feast-on-kubernetes/getting-started/README.md)
-  * [Install Feast](feast-on-kubernetes/getting-started/install-feast/README.md)
-    * [Docker Compose](feast-on-kubernetes/getting-started/install-feast/quickstart.md)
-    * [Kubernetes \(with Helm\)](feast-on-kubernetes/getting-started/install-feast/kubernetes-with-helm.md)
-    * [Amazon EKS \(with Terraform\)](feast-on-kubernetes/getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md)
-    * [Azure AKS \(with Helm\)](feast-on-kubernetes/getting-started/install-feast/kubernetes-azure-aks-with-helm.md)
-    * [Azure AKS \(with Terraform\)](feast-on-kubernetes/getting-started/install-feast/kubernetes-azure-aks-with-terraform.md)
-    * [Google Cloud GKE \(with Terraform\)](feast-on-kubernetes/getting-started/install-feast/google-cloud-gke-with-terraform.md)
-    * [IBM Cloud Kubernetes Service \(IKS\) and Red Hat OpenShift \(with Kustomize\)](feast-on-kubernetes/getting-started/install-feast/ibm-cloud-iks-with-kustomize.md)
-  * [Connect to Feast](feast-on-kubernetes/getting-started/connect-to-feast/README.md)
-    * [Python SDK](feast-on-kubernetes/getting-started/connect-to-feast/python-sdk.md)
-    * [Feast CLI](feast-on-kubernetes/getting-started/connect-to-feast/feast-cli.md)
-  * [Learn Feast](feast-on-kubernetes/getting-started/learn-feast.md)
-* [Concepts](feast-on-kubernetes/concepts/README.md)
-  * [Overview](feast-on-kubernetes/concepts/overview.md)
-  * [Architecture](feast-on-kubernetes/concepts/architecture.md)
-  * [Entities](feast-on-kubernetes/concepts/entities.md)
-  * [Sources](feast-on-kubernetes/concepts/sources.md)
-  * [Feature Tables](feast-on-kubernetes/concepts/feature-tables.md)
-  * [Stores](feast-on-kubernetes/concepts/stores.md)
-* [Tutorials](feast-on-kubernetes/tutorials-1/README.md)
-  * [Minimal Ride Hailing Example](https://github.com/feast-dev/feast/blob/master/examples/minimal/minimal_ride_hailing.ipynb)
-* [User guide](feast-on-kubernetes/user-guide/README.md)
-  * [Overview](feast-on-kubernetes/user-guide/overview.md)
-  * [Getting online features](feast-on-kubernetes/user-guide/getting-online-features.md)
-  * [Getting training features](feast-on-kubernetes/user-guide/getting-training-features.md)
-  * [Define and ingest features](feast-on-kubernetes/user-guide/define-and-ingest-features.md)
-  * [Extending Feast](feast-on-kubernetes/user-guide/extending-feast.md)
-* [Reference](feast-on-kubernetes/reference-1/README.md)
-  * [Configuration Reference](feast-on-kubernetes/reference-1/configuration-reference.md)
-  * [Feast and Spark](feast-on-kubernetes/reference-1/feast-and-spark.md)
-  * [Metrics Reference](feast-on-kubernetes/reference-1/metrics-reference.md)
-  * [Limitations](feast-on-kubernetes/reference-1/limitations.md)
-  * [API Reference](feast-on-kubernetes/reference-1/api/README.md)
-    * [Go SDK](https://godoc.org/github.com/feast-dev/feast/sdk/go)
-    * [Java SDK](https://javadoc.io/doc/dev.feast/feast-sdk)
-    * [Core gRPC API](https://api.docs.feast.dev/grpc/feast/core/coreservice.pb.html)
-    * [Python SDK](https://api.docs.feast.dev/python/)
-    * [Serving gRPC API](https://api.docs.feast.dev/grpc/feast/serving/servingservice.pb.html)
-    * [gRPC Types](https://api.docs.feast.dev/grpc/feast/types/value.pb)
-    * [Online Store Format Spec](specs/online_store_format.md)
-    * [Offline Store Format Spec](specs/offline_store_format.md)
-* [Advanced](feast-on-kubernetes/advanced-1/README.md)
-  * [Troubleshooting](feast-on-kubernetes/advanced-1/troubleshooting.md)
-  * [Metrics](feast-on-kubernetes/advanced-1/metrics.md)
-  * [Audit Logging](feast-on-kubernetes/advanced-1/audit-logging.md)
-  * [Security](feast-on-kubernetes/advanced-1/security.md)
-  * [Upgrading Feast](feast-on-kubernetes/advanced-1/upgrading.md)
-
 ## Contributing
 
 * [Contribution process](contributing/contributing.md)
 * [Development guide](contributing/development-guide.md)
 * [Versioning policy](contributing/versioning-policy.md)
 * [Release process](contributing/release-process.md)
 
+---
+
+* [Feast on Kubernetes](feast-on-kubernetes-1/README.md)
+  * [Getting started](feast-on-kubernetes-1/getting-started/README.md)
+    * [Install Feast](feast-on-kubernetes-1/getting-started/install-feast/README.md)
+      * [Docker Compose](feast-on-kubernetes-1/getting-started/install-feast/quickstart.md)
+      * [Kubernetes \(with Helm\)](feast-on-kubernetes-1/getting-started/install-feast/kubernetes-with-helm.md)
+      * [Amazon EKS \(with Terraform\)](feast-on-kubernetes-1/getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md)
+      * [Azure AKS \(with Helm\)](feast-on-kubernetes-1/getting-started/install-feast/kubernetes-azure-aks-with-helm.md)
+      * [Azure AKS \(with Terraform\)](feast-on-kubernetes-1/getting-started/install-feast/kubernetes-azure-aks-with-terraform.md)
+      * [Google Cloud GKE \(with Terraform\)](feast-on-kubernetes-1/getting-started/install-feast/google-cloud-gke-with-terraform.md)
+      * [IBM Cloud Kubernetes Service \(IKS\) and Red Hat OpenShift \(with Kustomize\)](feast-on-kubernetes-1/getting-started/install-feast/ibm-cloud-iks-with-kustomize.md)
+    * [Connect to Feast](feast-on-kubernetes-1/getting-started/connect-to-feast/README.md)
+      * [Python SDK](feast-on-kubernetes-1/getting-started/connect-to-feast/python-sdk.md)
+      * [Feast CLI](feast-on-kubernetes-1/getting-started/connect-to-feast/feast-cli.md)
+    * [Learn Feast](feast-on-kubernetes-1/getting-started/learn-feast.md)
+  * [Concepts](feast-on-kubernetes-1/concepts/README.md)
+    * [Overview](feast-on-kubernetes-1/concepts/overview.md)
+    * [Architecture](feast-on-kubernetes-1/concepts/architecture.md)
+    * [Entities](feast-on-kubernetes-1/concepts/entities.md)
+    * [Sources](feast-on-kubernetes-1/concepts/sources.md)
+    * [Feature Tables](feast-on-kubernetes-1/concepts/feature-tables.md)
+    * [Stores](feast-on-kubernetes-1/concepts/stores.md)
+  * [Tutorials](feast-on-kubernetes-1/tutorials-1/README.md)
+    * [Minimal Ride Hailing Example](https://github.com/feast-dev/feast/blob/master/examples/minimal/minimal_ride_hailing.ipynb)
+  * [User guide](feast-on-kubernetes-1/user-guide/README.md)
+    * [Overview](feast-on-kubernetes-1/user-guide/overview.md)
+    * [Getting online features](feast-on-kubernetes-1/user-guide/getting-online-features.md)
+    * [Getting training features](feast-on-kubernetes-1/user-guide/getting-training-features.md)
+    * [Define and ingest features](feast-on-kubernetes-1/user-guide/define-and-ingest-features.md)
+    * [Extending Feast](feast-on-kubernetes-1/user-guide/extending-feast.md)
+  * [Reference](feast-on-kubernetes-1/reference-1/README.md)
+    * [Configuration Reference](feast-on-kubernetes-1/reference-1/configuration-reference.md)
+    * [Feast and Spark](feast-on-kubernetes-1/reference-1/feast-and-spark.md)
+    * [Metrics Reference](feast-on-kubernetes-1/reference-1/metrics-reference.md)
+    * [Limitations](feast-on-kubernetes-1/reference-1/limitations.md)
+    * [API Reference](feast-on-kubernetes-1/reference-1/api/README.md)
+      * [Go SDK](https://godoc.org/github.com/feast-dev/feast/sdk/go)
+      * [Java SDK](https://javadoc.io/doc/dev.feast/feast-sdk)
+      * [Core gRPC API](https://api.docs.feast.dev/grpc/feast/core/coreservice.pb.html)
+      * [Python SDK](https://api.docs.feast.dev/python/)
+      * [Serving gRPC API](https://api.docs.feast.dev/grpc/feast/serving/servingservice.pb.html)
+      * [gRPC Types](https://api.docs.feast.dev/grpc/feast/types/value.pb)
+      * [Online Store Format Spec](feast-on-kubernetes-1/reference-1/api/online_store_format.md)
+      * [Offline Store Format Spec](feast-on-kubernetes-1/reference-1/api/offline_store_format.md)
+  * [Advanced](feast-on-kubernetes-1/advanced-1/README.md)
+    * [Troubleshooting](feast-on-kubernetes-1/advanced-1/troubleshooting.md)
+    * [Metrics](feast-on-kubernetes-1/advanced-1/metrics.md)
+    * [Audit Logging](feast-on-kubernetes-1/advanced-1/audit-logging.md)
+    * [Security](feast-on-kubernetes-1/advanced-1/security.md)
+    * [Upgrading Feast](feast-on-kubernetes-1/advanced-1/upgrading.md)
+

docs/.gitbook/assets/high_level_hierarchy_redis.png

@@ -28,6 +28,6 @@ A complete Feast deployment contains the following components:
 * **Offline Store:** The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. Feast does not manage the offline store directly, but runs queries against it.
 
 {% hint style="info" %}
-Java and Go Clients are also available for online feature retrieval. See [API Reference](../feast-on-kubernetes/reference-1/api/).
+Java and Go Clients are also available for online feature retrieval. See [API Reference](../feast-on-kubernetes-1/reference-1/api/).
 {% endhint %}
 

docs/.gitbook/assets/redis_online_example.png

@@ -22,7 +22,7 @@ For Feast maintainers, these are the concrete steps for making a new release.
    2. Add the change log by applying the change log commit created in step 2.
    3. Check that versions are updated with `env TARGET_MERGE_BRANCH=master make lint-versions`
 7. Create a [GitHub release](https://github.com/feast-dev/feast/releases) which includes a summary of im~~p~~ortant changes as well as any artifacts associated with the release. Make sure to include the same change log as added in [CHANGELOG.md](https://github.com/feast-dev/feast/blob/master/CHANGELOG.md). Use `Feast vX.Y.Z` as the title.
-8. Update the[ Upgrade Guide](../feast-on-kubernetes/advanced-1/upgrading.md)  to include the action required instructions for users to upgrade to this new release. Instructions should include a migration for each breaking change made to this release.
+8. Update the[ Upgrade Guide](../feast-on-kubernetes-1/advanced-1/upgrading.md)  to include the action required instructions for users to upgrade to this new release. Instructions should include a migration for each breaking change made to this release.
 
 When a tag that matches a Semantic Version string is pushed, CI will automatically build and push the relevant artifacts to their repositories or package managers \(docker images, Python wheels, etc\). JVM artifacts are promoted from Sonatype OSSRH to Maven Central, but it sometimes takes some time for them to be available. The `sdk/go/v tag` is required to version the Go SDK go module so that users can go get a specific tagged release of the Go SDK.
 

docs/SUMMARY.md

@@ -0,0 +1,8 @@
+# Feast on Kubernetes
+
+{% hint style="success" %}
+Since Feast 0.10, it is not necessary to deploy Feast into Kubernetes any more. Feast runs as a light weight CLI/SDK that can configure your managed services to operate as a feature store. The documentation contained within this section exists for posterity sake and relates to Feast 0.9, which depends on both Kubernetes and Spark. Teams that want to consume from streaming sources have to continue to use this release until streaming support has been reintroduced to Feast.
+{% endhint %}
+
+
+

docs/concepts/architecture-and-components.md

@@ -0,0 +1,2 @@
+# Advanced
+

docs/contributing/release-process.md

@@ -0,0 +1,132 @@
+# Audit Logging
+
+{% hint style="warning" %}
+This page applies to Feast 0.7. The content may be out of date for Feast 0.8+
+{% endhint %}
+
+## Introduction
+
+Feast provides audit logging functionality in order to debug problems and to trace the lineage of events.
+
+## Audit Log Types
+
+Audit Logs produced by Feast come in three favors:
+
+| Audit Log Type | Description |
+| :--- | :--- |
+| Message Audit Log | Logs service calls that can be used to track Feast request handling. Currently only gRPC request/response is supported. Enabling Message Audit Logs can be resource intensive and significantly increase latency, as such is not recommended on Online Serving. |
+| Transition Audit Log | Logs transitions in status in resources managed by Feast \(ie an Ingestion Job becoming RUNNING\). |
+| Action Audit Log | Logs actions performed on a specific resource managed by Feast \(ie an Ingestion Job is aborted\). |
+
+## Configuration
+
+| Audit Log Type | Description |
+| :--- | :--- |
+| Message Audit Log | Enabled when both `feast.logging.audit.enabled` and `feast.logging.audit.messageLogging.enabled` is set to `true` |
+| Transition Audit Log | Enabled when `feast.logging.audit.enabled` is set to `true` |
+| Action Audit Log | Enabled when `feast.logging.audit.enabled` is set to `true` |
+
+## JSON Format
+
+Audit Logs produced by Feast are written to the console similar to normal logs but in a structured, machine parsable JSON. Example of a Message Audit Log JSON entry produced:
+
+```text
+{
+  "message": {
+    "logType": "FeastAuditLogEntry",
+    "kind": "MESSAGE",
+    "statusCode": "OK",
+    "request": {
+      "filter": {
+        "project": "dummy",
+      }
+    },
+    "application": "Feast",
+    "response": {},
+    "method": "ListFeatureTables",
+    "identity": "105960238928959148073",
+    "service": "CoreService",
+    "component": "feast-core",
+    "id": "45329ea9-0d48-46c5-b659-4604f6193711",
+    "version": "0.10.0-SNAPSHOT"
+  },
+  "hostname": "feast.core"
+  "timestamp": "2020-10-20T04:45:24Z",
+  "severity": "INFO",
+}
+```
+
+## Log Entry Schema
+
+Fields common to all Audit Log Types:
+
+| Field | Description |
+| :--- | :--- |
+| `logType` | Log Type. Always set to `FeastAuditLogEntry`. Useful for filtering out Feast audit logs. |
+| `application` | Application. Always set to `Feast`. |
+| `component` | Feast Component producing the Audit Log. Set to `feast-core` for Feast Core and `feast-serving` for Feast Serving. Use to filtering out Audit Logs by component. |
+| `version` | Version of Feast producing this Audit Log. Use to filtering out Audit Logs by version. |
+
+Fields in Message Audit Log Type
+
+| Field | Description |
+| :--- | :--- |
+| `id` | Generated UUID that uniquely identifies the service call. |
+| `service` | Name of the Service that handled the service call. |
+| `method` | Name of the Method that handled the service call. Useful for filtering Audit Logs by method \(ie `ApplyFeatureTable` calls\) |
+| `request` | Full request submitted by client in the service call as JSON. |
+| `response` | Full response returned to client by the service after handling the service call as JSON. |
+| `identity` | Identity of the client making the service call as an user Id. Only set when Authentication is enabled. |
+| `statusCode` | The status code returned by the service handling the service call \(ie `OK` if service call handled without error\). |
+
+Fields in Action Audit Log Type
+
+| Field | Description |
+| :--- | :--- |
+| `action` | Name of the action taken on the resource. |
+| `resource.type` | Type of resource of which the action was taken on \(i.e `FeatureTable`\) |
+| resource.id | Identifier specifying the specific resource of which the action was taken on. |
+
+Fields in Transition Audit Log Type
+
+| Field | Description |
+| :--- | :--- |
+| `status` | The new status that the resource transitioned to |
+| `resource.type` | Type of resource of which the transition occurred \(i.e `FeatureTable`\) |
+| `resource.id` | Identifier specifying the specific resource of which the transition occurred. |
+
+## Log Forwarder
+
+Feast currently only supports forwarding Request/Response \(Message Audit Log Type\) logs to an external fluentD service with `feast.**` Fluentd tag.
+
+### Request/Response Log Example
+
+```text
+{
+  "id": "45329ea9-0d48-46c5-b659-4604f6193711",
+  "service": "CoreService"
+  "status_code": "OK",
+  "identity": "105960238928959148073",
+  "method": "ListProjects",
+  "request": {},
+  "response": {
+    "projects": [
+      "default", "project1", "project2"
+    ]
+  }
+  "release_name": 506.457.14.512
+}
+```
+
+### Configuration
+
+The Fluentd Log Forwarder configured with the with the following configuration options in `application.yml`:
+
+| Settings | Description |
+| :--- | :--- |
+| `feast.logging.audit.messageLogging.destination` | `fluentd` |
+| `feast.logging.audit.messageLogging.fluentdHost` | `localhost` |
+| `feast.logging.audit.messageLogging.fluentdPort` | `24224` |
+
+When using Fluentd as the Log forwarder, a Feast `release_name` can be logged instead of the IP address \(eg. IP of Kubernetes pod deployment\), by setting an environment variable `RELEASE_NAME` when deploying Feast.
+

docs/feast-on-kubernetes-1/README.md

@@ -0,0 +1,59 @@
+# Metrics
+
+{% hint style="warning" %}
+This page applies to Feast 0.7. The content may be out of date for Feast 0.8+
+{% endhint %}
+
+## Overview
+
+Feast Components export metrics that can provide insight into Feast behavior:
+
+* [Feast Ingestion Jobs  can be configured to push metrics into StatsD](metrics.md#pushing-ingestion-metrics-to-statsd)
+* [Prometheus can be configured to scrape metrics from Feast Core and Serving.](metrics.md#exporting-feast-metrics-to-prometheus)
+
+See the [Metrics Reference ](../reference-1/metrics-reference.md)for documentation on metrics are exported by Feast.
+
+{% hint style="info" %}
+Feast Job Controller currently does not export any metrics on its own. However its `application.yml` is used to configure metrics export for ingestion jobs.
+{% endhint %}
+
+## Pushing Ingestion Metrics to StatsD
+
+### **Feast Ingestion Job**
+
+Feast Ingestion Job can be configured to push Ingestion metrics to a StatsD instance. Metrics export to StatsD for Ingestion Job is configured in Job Controller's `application.yml` under `feast.jobs.metrics`
+
+```yaml
+ feast:
+   jobs:
+    metrics:
+      # Enables Statd metrics export if true.
+      enabled: true
+      type: statsd
+      # Host and port of the StatsD instance to export to.
+      host: localhost
+      port: 9125
+```
+
+{% hint style="info" %}
+If you need Ingestion Metrics in Prometheus or some other metrics backend, use a metrics forwarder to forward Ingestion Metrics from StatsD to the metrics backend of choice. \(ie Use [`prometheus-statsd-exporter`](https://github.com/prometheus/statsd_exporter) to forward metrics to Prometheus\).
+{% endhint %}
+
+## Exporting Feast Metrics to Prometheus
+
+### **Feast Core and Serving**
+
+Feast Core and Serving exports metrics to a Prometheus instance via Prometheus scraping its `/metrics` endpoint. Metrics export to Prometheus for Core and Serving can be configured via their corresponding `application.yml`
+
+```yaml
+server:
+  # Configures the port where metrics are exposed via /metrics for Prometheus to scrape.
+  port: 8081
+```
+
+[Direct Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) to scrape directly from Core and Serving's `/metrics` endpoint.
+
+## Further Reading
+
+See the [Metrics Reference ](../reference-1/metrics-reference.md)for documentation on metrics are exported by Feast.
+