Data Enrichment using Research API and formatting with GPT function calling #8
Conversation
asleshapokhrel-georgian
requested review from
rmcarthur,
rodrigo-georgian,
royal-gp and
davidli3100
and removed request for
rmcarthur
| "# obtain the API key from the environment\n", | ||
| "headers = {'x-api-key': os.environ['YDC_API_KEY']}\n", | ||
| "\n", | ||
| "def get_research_data(company_name, missing_cols, mode=\"research\"):\n", |
There was a problem hiding this comment.
@christeefy, what's the latest recommended way to do this? Can they import a retriever instead here?
There was a problem hiding this comment.
Retriever implementations are framework-specific. They can import a retriever — the question becomes more about whether we want this example to be framework-agnostic or not. If yes to being framework-agnostic, then pinging the API directly would be the approach.
There was a problem hiding this comment.
Oh my bad, this is the chat API, which is implemented as an LLM (instead of a retriever). It is still framework-specific, so my comments above still apply.
There was a problem hiding this comment.
Do you know which retrievers have been merged already?
We have examples for langchain and llama index. Any others that they could use here to show off what we have done? If not, I'd suggest going with the llama-index one as we only have one example for that one right now.
There was a problem hiding this comment.
@rodrigo-georgian is llama-index using research API? -> To answer my own question - It's using the search API
There was a problem hiding this comment.
@rodrigo-georgian All framework's You retrievers support the search API. Langchain's support News as well.
PRs have been put up across all frameworks to update all their implementations to support News but not yet merged by the authors.
There was a problem hiding this comment.
PRs for the chat API have not been created until YDC is ready.
There was a problem hiding this comment.
Discussed with @rodrigo-georgian offline. This example showcases research API. We will leave the implementation as is for now to use rest api and switch with langchain/dspy framework once available.
This notebook shows the guide to enrich missing tabular data for a sample of synthetic data.
You.comresearch API is used to get all the relevant data from web and GPT function calling is used to extract the relevant information in a specific format which is used to populate the tables.