Skip to content

perf: inline read_pandas for small data #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Feb 21, 2024
Merged

perf: inline read_pandas for small data #383

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
912b874
perf: inline read_pandas for small data
GarrettWu Feb 16, 2024
22dc85c
fix tests
GarrettWu Feb 16, 2024
e113de7
revert unrelated file
GarrettWu Feb 16, 2024
3a8520a
fix tests
GarrettWu Feb 16, 2024
b2e003b
fix tests
GarrettWu Feb 16, 2024
912af6e
fix tests
GarrettWu Feb 20, 2024
b533641
fix tests
GarrettWu Feb 20, 2024
86f7eeb
fix tests
GarrettWu Feb 20, 2024
d373770
fix tests
GarrettWu Feb 20, 2024
abd65b9
fix tests
GarrettWu Feb 20, 2024
85ff23a
Merge branch 'main' into garrettwu-read
GarrettWu Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tests
  • Loading branch information
@GarrettWu
GarrettWu committed Feb 16, 2024
commit 22dc85c047160beca727258084434bae269a5b64
214 changes: 213 additions & 1 deletion notebooks/getting_started/getting_started_bq_dataframes.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"id": "PyQmSRbKA8r-"
},
Expand All @@ -365,6 +365,15 @@
"### Set BigQuery DataFrames options"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -377,6 +386,81 @@
"bf.options.bigquery.location = REGION"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Query job 4dc6c41c-3e96-47db-98e4-ae6c2f6bf50e is DONE. 0 Bytes processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=bigframes-dev&j=bq:US:4dc6c41c-3e96-47db-98e4-ae6c2f6bf50e&page=queryresults\">Open Job</a>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>2 rows × 2 columns</p>\n",
"</div>[2 rows x 2 columns in total]"
],
"text/plain": [
" 1 2\n",
"0 1 3\n",
"1 2 4\n",
"\n",
"[2 rows x 2 columns]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bf.DataFrame({\"1\": [1, 2], \"2\": [3, 4]})"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand All @@ -386,6 +470,122 @@
"If you want to reset the location of the created DataFrame or Series objects, reset the session by executing `bf.close_session()`. After that, you can reuse `bf.options.bigquery.location` to specify another location."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"pd_df = pd.DataFrame({\"1\": [1, 2], \"2\": [3, 4]})"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Load job c9eedc60-a8ca-4b5c-8043-fd38525b5126 is DONE. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=bigframes-dev&j=bq:US:c9eedc60-a8ca-4b5c-8043-fd38525b5126&page=queryresults\">Open Job</a>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/google/home/garrettwu/.pyenv/versions/3.10.9/lib/python3.10/site-packages/ibis/backends/bigquery/__init__.py:499: FutureWarning: `database` is deprecated as of v7.1, removed in v8.0; The bigquery backend cannot return a table expression using only a `database` specifier. Include a `schema` argument.\n",
" util.warn_deprecated(\n"
]
},
{
"data": {
"text/html": [
"Query job e9c219ee-880f-464a-9464-cb2dab20d498 is DONE. 0 Bytes processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=bigframes-dev&j=bq:US:e9c219ee-880f-464a-9464-cb2dab20d498&page=queryresults\">Open Job</a>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Query job d852b9ab-3814-4fd0-91af-53c57152c027 is DONE. 64 Bytes processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=bigframes-dev&j=bq:US:d852b9ab-3814-4fd0-91af-53c57152c027&page=queryresults\">Open Job</a>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>2 rows × 2 columns</p>\n",
"</div>[2 rows x 2 columns in total]"
],
"text/plain": [
" 1 2\n",
"0 1 3\n",
"1 2 4\n",
"\n",
"[2 rows x 2 columns]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bf.read_pandas(pd_df)"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -973,6 +1173,18 @@
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
Expand Down
3 changes: 1 addition & 2 deletions tests/unit/session/test_io_bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,8 @@ def test_create_job_configs_labels_log_adaptor_call_method_under_length_limit():
"recent-bigframes-api-2": "dataframe-__init__",
"recent-bigframes-api-3": "dataframe-head",
"recent-bigframes-api-4": "dataframe-__init__",
"recent-bigframes-api-5": "dataframe-__init__",
}
assert labels is not None
assert len(labels) == 7
assert labels == expected_dict


Expand Down