Skip to content

DATE converted to dbdate instead of datetime64[ns] dtype #579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ivan-toriya-precis opened this issue Oct 19, 2022 · 2 comments · Fixed by #598
Closed

DATE converted to dbdate instead of datetime64[ns] dtype #579

ivan-toriya-precis opened this issue Oct 19, 2022 · 2 comments · Fixed by #598
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: cleanup An internal cleanup or hygiene concern.

Comments

@ivan-toriya-precis
Copy link

Environment details

  • OS type and version:
    Env: Google Colab
    OS type: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
    Version: #1 SMP Fri Aug 26 08:44:51 UTC 2022

  • Python version: python --version
    Python 3.7.15

  • pip version: pip --version
    pip 21.1.3 from /usr/local/lib/python3.7/dist-packages/pip (python 3.7)

  • pandas-gbq version: pip show pandas-gbq

Name: pandas-gbq
Version: 0.17.9
Summary: Google BigQuery connector for pandas
Home-page: https://github.com/googleapis/python-bigquery-pandas
Author: pandas-gbq authors
Author-email: [[email protected]](mailto:[email protected])
License: BSD-3-Clause
Location: /usr/local/lib/python3.7/dist-packages
Requires: db-dtypes, google-api-core, google-auth, google-auth-oauthlib, google-cloud-bigquery, google-cloud-bigquery-storage, numpy, pandas, pyarrow, pydata-google-auth, setuptools
Required-by:

Steps to reproduce

  1. Query any table with column DATE BigQuery type.
  2. Expect that the column will be converted to datetime64[ns] pandas dtype as said in the docs.
  3. In reality it gets converted to dbdate type.

Code example

import pandas_gbq
from google.colab import auth

auth.authenticate_user()

project_id = ""  # TODO: provide project ID

sql = f'''
  SELECT
    *
  FROM
    `bigquery-public-data.google_cloud_release_notes.release_notes`
  WHERE
    RAND() <= 0.1
'''

df = pandas_gbq.read_gbq(
    sql,
    project_id=project_id)

df.info()
Downloading: 100%|██████████| 1543/1543 [00:00<00:00, 3235.38rows/s]<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1543 entries, 0 to 1542
Data columns (total 6 columns):
 #   Column                Non-Null Count  Dtype 
---  ------                --------------  ----- 
 0   description           1543 non-null   object
 1   release_note_type     1543 non-null   object
 2   published_at          1543 non-null   dbdate
 3   product_id            1543 non-null   Int64 
 4   product_name          1543 non-null   object
 5   product_version_name  1091 non-null   object
dtypes: Int64(1), dbdate(1), object(4)
memory usage: 74.0+ KB
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label Oct 19, 2022
@tswast
Copy link
Collaborator

tswast commented Nov 18, 2022

Thanks for the report. I think dbdate is the dtype we'll want for these columns. We should update the documentation.

@tswast
Copy link
Collaborator

tswast commented Nov 28, 2022

Related: readthedocs is out-of-date #448

But https://googleapis.dev/python/pandas-gbq/latest/reading.html#inferring-the-dataframe-s-dtypes also needs an update.

@tswast tswast added the type: cleanup An internal cleanup or hygiene concern. label Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants