Skip to content

Commit be004ef

Browse files
ntkatholentkathole
authored
feat: Added Lineage APIs to get registry objects relationships (feast-dev#5472)
* feat: Added Lineage APIs to get registry objects relationships Signed-off-by: ntkathole <[email protected]> * docs: Added docs for rest api endpoints Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]>
1 parent 10879ec commit be004ef

File tree

15 files changed

+1686
-6
lines changed

15 files changed

+1686
-6
lines changed

‎docs/reference/feature-servers/registry-server.md‎

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,276 @@ Feast supports running the Registry Server in three distinct modes:
1212
| REST + gRPC | `feast serve_registry --rest-api` | Enables both interfaces |
1313
| REST only | `feast serve_registry --rest-api --no-grpc` | Used for REST-only clients like the UI |
1414

15+
## REST API Endpoints
16+
17+
The REST API provides HTTP/JSON endpoints for accessing all registry metadata. All endpoints are prefixed with `/api/v1` and return JSON responses.
18+
19+
### Authentication
20+
21+
The REST API supports Bearer token authentication. Include your token in the Authorization header:
22+
23+
```bash
24+
Authorization: Bearer <your-token>
25+
```
26+
27+
### Common Query Parameters
28+
29+
Most endpoints support these common query parameters:
30+
31+
- `project` (required for most endpoints): The project name
32+
- `allow_cache` (optional, default: `true`): Whether to allow cached data
33+
- `tags` (optional): Filter results by tags in key=value format
34+
35+
### Entities
36+
37+
#### List Entities
38+
- **Endpoint**: `GET /api/v1/entities`
39+
- **Description**: Retrieve all entities in a project
40+
- **Parameters**:
41+
- `project` (required): Project name
42+
- **Example**:
43+
```bash
44+
curl -H "Authorization: Bearer <token>" \
45+
"http://localhost:6572/api/v1/entities?project=my_project"
46+
```
47+
48+
#### Get Entity
49+
- **Endpoint**: `GET /api/v1/entities/{name}`
50+
- **Description**: Retrieve a specific entity by name
51+
- **Parameters**:
52+
- `name` (path): Entity name
53+
- `project` (required): Project name
54+
- `allow_cache` (optional): Whether to allow cached data
55+
- **Example**:
56+
```bash
57+
curl -H "Authorization: Bearer <token>" \
58+
"http://localhost:6572/api/v1/entities/user_id?project=my_project"
59+
```
60+
61+
### Data Sources
62+
63+
#### List Data Sources
64+
- **Endpoint**: `GET /api/v1/data_sources`
65+
- **Description**: Retrieve all data sources in a project
66+
- **Parameters**:
67+
- `project` (required): Project name
68+
- `allow_cache` (optional): Whether to allow cached data
69+
- `tags` (optional): Filter by tags
70+
- **Example**:
71+
```bash
72+
curl -H "Authorization: Bearer <token>" \
73+
"http://localhost:6572/api/v1/data_sources?project=my_project"
74+
```
75+
76+
#### Get Data Source
77+
- **Endpoint**: `GET /api/v1/data_sources/{name}`
78+
- **Description**: Retrieve a specific data source by name
79+
- **Parameters**:
80+
- `name` (path): Data source name
81+
- `project` (required): Project name
82+
- `allow_cache` (optional): Whether to allow cached data
83+
- **Example**:
84+
```bash
85+
curl -H "Authorization: Bearer <token>" \
86+
"http://localhost:6572/api/v1/data_sources/user_data?project=my_project"
87+
```
88+
89+
### Feature Views
90+
91+
#### List Feature Views
92+
- **Endpoint**: `GET /api/v1/feature_views`
93+
- **Description**: Retrieve all feature views in a project
94+
- **Parameters**:
95+
- `project` (required): Project name
96+
- `allow_cache` (optional): Whether to allow cached data
97+
- `tags` (optional): Filter by tags
98+
- **Example**:
99+
```bash
100+
curl -H "Authorization: Bearer <token>" \
101+
"http://localhost:6572/api/v1/feature_views?project=my_project"
102+
```
103+
104+
#### Get Feature View
105+
- **Endpoint**: `GET /api/v1/feature_views/{name}`
106+
- **Description**: Retrieve a specific feature view by name
107+
- **Parameters**:
108+
- `name` (path): Feature view name
109+
- `project` (required): Project name
110+
- `allow_cache` (optional): Whether to allow cached data
111+
- **Example**:
112+
```bash
113+
curl -H "Authorization: Bearer <token>" \
114+
"http://localhost:6572/api/v1/feature_views/user_features?project=my_project"
115+
```
116+
117+
### Feature Services
118+
119+
#### List Feature Services
120+
- **Endpoint**: `GET /api/v1/feature_services`
121+
- **Description**: Retrieve all feature services in a project
122+
- **Parameters**:
123+
- `project` (required): Project name
124+
- `allow_cache` (optional): Whether to allow cached data
125+
- `tags` (optional): Filter by tags
126+
- **Example**:
127+
```bash
128+
curl -H "Authorization: Bearer <token>" \
129+
"http://localhost:6572/api/v1/feature_services?project=my_project"
130+
```
131+
132+
#### Get Feature Service
133+
- **Endpoint**: `GET /api/v1/feature_services/{name}`
134+
- **Description**: Retrieve a specific feature service by name
135+
- **Parameters**:
136+
- `name` (path): Feature service name
137+
- `project` (required): Project name
138+
- `allow_cache` (optional): Whether to allow cached data
139+
- **Example**:
140+
```bash
141+
curl -H "Authorization: Bearer <token>" \
142+
"http://localhost:6572/api/v1/feature_services/recommendation_service?project=my_project"
143+
```
144+
145+
### Lineage and Relationships
146+
147+
#### Get Registry Lineage
148+
- **Endpoint**: `GET /api/v1/lineage/registry`
149+
- **Description**: Retrieve complete registry lineage with relationships and indirect relationships
150+
- **Parameters**:
151+
- `project` (required): Project name
152+
- `allow_cache` (optional): Whether to allow cached data
153+
- `filter_object_type` (optional): Filter by object type (`dataSource`, `entity`, `featureView`, `featureService`)
154+
- `filter_object_name` (optional): Filter by object name
155+
- **Example**:
156+
```bash
157+
curl -H "Authorization: Bearer <token>" \
158+
"http://localhost:6572/api/v1/lineage/registry?project=my_project"
159+
```
160+
161+
#### Get Object Relationships
162+
- **Endpoint**: `GET /api/v1/lineage/objects/{object_type}/{object_name}`
163+
- **Description**: Retrieve relationships for a specific object
164+
- **Parameters**:
165+
- `object_type` (path): Type of object (`dataSource`, `entity`, `featureView`, `featureService`)
166+
- `object_name` (path): Name of the object
167+
- `project` (required): Project name
168+
- `include_indirect` (optional): Whether to include indirect relationships
169+
- `allow_cache` (optional): Whether to allow cached data
170+
- **Example**:
171+
```bash
172+
curl -H "Authorization: Bearer <token>" \
173+
"http://localhost:6572/api/v1/lineage/objects/featureView/user_features?project=my_project&include_indirect=true"
174+
```
175+
176+
#### Get Complete Registry Data
177+
- **Endpoint**: `GET /api/v1/lineage/complete`
178+
- **Description**: Retrieve complete registry data including all objects and relationships (optimized for UI consumption)
179+
- **Parameters**:
180+
- `project` (required): Project name
181+
- `allow_cache` (optional): Whether to allow cached data
182+
- **Example**:
183+
```bash
184+
curl -H "Authorization: Bearer <token>" \
185+
"http://localhost:6572/api/v1/lineage/complete?project=my_project"
186+
```
187+
188+
### Permissions
189+
190+
#### List Permissions
191+
- **Endpoint**: `GET /api/v1/permissions`
192+
- **Description**: Retrieve all permissions in a project
193+
- **Parameters**:
194+
- `project` (required): Project name
195+
- `allow_cache` (optional): Whether to allow cached data
196+
- **Example**:
197+
```bash
198+
curl -H "Authorization: Bearer <token>" \
199+
"http://localhost:6572/api/v1/permissions?project=my_project"
200+
```
201+
202+
#### Get Permission
203+
- **Endpoint**: `GET /api/v1/permissions/{name}`
204+
- **Description**: Retrieve a specific permission by name
205+
- **Parameters**:
206+
- `name` (path): Permission name
207+
- `project` (required): Project name
208+
- `allow_cache` (optional): Whether to allow cached data
209+
- **Example**:
210+
```bash
211+
curl -H "Authorization: Bearer <token>" \
212+
"http://localhost:6572/api/v1/permissions/read_features?project=my_project"
213+
```
214+
215+
### Projects
216+
217+
#### List Projects
218+
- **Endpoint**: `GET /api/v1/projects`
219+
- **Description**: Retrieve all projects
220+
- **Parameters**:
221+
- `allow_cache` (optional): Whether to allow cached data
222+
- **Example**:
223+
```bash
224+
curl -H "Authorization: Bearer <token>" \
225+
"http://localhost:6572/api/v1/projects"
226+
```
227+
228+
#### Get Project
229+
- **Endpoint**: `GET /api/v1/projects/{name}`
230+
- **Description**: Retrieve a specific project by name
231+
- **Parameters**:
232+
- `name` (path): Project name
233+
- `allow_cache` (optional): Whether to allow cached data
234+
- **Example**:
235+
```bash
236+
curl -H "Authorization: Bearer <token>" \
237+
"http://localhost:6572/api/v1/projects/my_project"
238+
```
239+
240+
### Saved Datasets
241+
242+
#### List Saved Datasets
243+
- **Endpoint**: `GET /api/v1/saved_datasets`
244+
- **Description**: Retrieve all saved datasets in a project
245+
- **Parameters**:
246+
- `project` (required): Project name
247+
- `allow_cache` (optional): Whether to allow cached data
248+
- `tags` (optional): Filter by tags
249+
- **Example**:
250+
```bash
251+
curl -H "Authorization: Bearer <token>" \
252+
"http://localhost:6572/api/v1/saved_datasets?project=my_project"
253+
```
254+
255+
#### Get Saved Dataset
256+
- **Endpoint**: `GET /api/v1/saved_datasets/{name}`
257+
- **Description**: Retrieve a specific saved dataset by name
258+
- **Parameters**:
259+
- `name` (path): Saved dataset name
260+
- `project` (required): Project name
261+
- `allow_cache` (optional): Whether to allow cached data
262+
- **Example**:
263+
```bash
264+
curl -H "Authorization: Bearer <token>" \
265+
"http://localhost:6572/api/v1/saved_datasets/training_data?project=my_project"
266+
```
267+
268+
### Response Formats
269+
270+
All endpoints return JSON responses with the following general structure:
271+
272+
- **Success (200)**: Returns the requested data
273+
- **Bad Request (400)**: Invalid parameters or request format
274+
- **Unauthorized (401)**: Missing or invalid authentication token
275+
- **Not Found (404)**: Requested resource does not exist
276+
- **Internal Server Error (500)**: Server-side error
277+
278+
### Interactive API Documentation
279+
280+
When the REST API server is running, you can access interactive documentation at:
281+
282+
- **Swagger UI**: `http://localhost:6572/` (root path)
283+
- **ReDoc**: `http://localhost:6572/docs`
284+
- **OpenAPI Schema**: `http://localhost:6572/openapi.json`
15285

16286
## How to configure the server
17287

‎protos/feast/registry/RegistryServer.proto‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,10 @@ service RegistryServer{
8888
rpc Refresh (RefreshRequest) returns (google.protobuf.Empty) {}
8989
rpc Proto (google.protobuf.Empty) returns (feast.core.Registry) {}
9090

91+
// Lineage RPCs
92+
rpc GetRegistryLineage (GetRegistryLineageRequest) returns (GetRegistryLineageResponse) {}
93+
rpc GetObjectRelationships (GetObjectRelationshipsRequest) returns (GetObjectRelationshipsResponse) {}
94+
9195
}
9296

9397
message RefreshRequest {
@@ -424,3 +428,39 @@ message DeleteProjectRequest {
424428
string name = 1;
425429
bool commit = 2;
426430
}
431+
432+
// Lineage
433+
434+
message EntityReference {
435+
string type = 1; // "dataSource", "entity", "featureView", "featureService"
436+
string name = 2;
437+
}
438+
439+
message EntityRelation {
440+
EntityReference source = 1;
441+
EntityReference target = 2;
442+
}
443+
444+
message GetRegistryLineageRequest {
445+
string project = 1;
446+
bool allow_cache = 2;
447+
string filter_object_type = 3;
448+
string filter_object_name = 4;
449+
}
450+
451+
message GetRegistryLineageResponse {
452+
repeated EntityRelation relationships = 1;
453+
repeated EntityRelation indirect_relationships = 2;
454+
}
455+
456+
message GetObjectRelationshipsRequest {
457+
string project = 1;
458+
string object_type = 2;
459+
string object_name = 3;
460+
bool include_indirect = 4;
461+
bool allow_cache = 5;
462+
}
463+
464+
message GetObjectRelationshipsResponse {
465+
repeated EntityRelation relationships = 1;
466+
}

‎sdk/python/feast/api/registry/rest/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
from feast.api.registry.rest.entities import get_entity_router
55
from feast.api.registry.rest.feature_services import get_feature_service_router
66
from feast.api.registry.rest.feature_views import get_feature_view_router
7+
from feast.api.registry.rest.lineage import get_lineage_router
78
from feast.api.registry.rest.permissions import get_permission_router
89
from feast.api.registry.rest.projects import get_project_router
910
from feast.api.registry.rest.saved_datasets import get_saved_dataset_router
@@ -14,6 +15,7 @@ def register_all_routes(app: FastAPI, grpc_handler):
1415
app.include_router(get_data_source_router(grpc_handler))
1516
app.include_router(get_feature_service_router(grpc_handler))
1617
app.include_router(get_feature_view_router(grpc_handler))
18+
app.include_router(get_lineage_router(grpc_handler))
1719
app.include_router(get_permission_router(grpc_handler))
1820
app.include_router(get_project_router(grpc_handler))
1921
app.include_router(get_saved_dataset_router(grpc_handler))

0 commit comments

Comments
 (0)