Skip to content

feature request: to_gbq with a bigframes.pandas.DataFrame should call .to_gbq() from BigQuery DataFrames not download and re-upload #824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tswast opened this issue Nov 5, 2024 · 1 comment · Fixed by #833
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Collaborator

tswast commented Nov 5, 2024

Is your feature request related to a problem? Please describe.

Mixing pandas-gbq and BigFrames can be a recipe for confusion and increased costs. Sending a bigframes.pandas.DataFrame to pandas_gbq.to_gbq results in many queries as pandas-gbq tries to determine the appropriate schema to use for upload.

Describe the solution you'd like

Ideally, if bigframes is installed, pandas-gbq can detect if someone is trying to use a BigQuery DataFrames object with it and call the .to_gbq() method on the object, which would avoid unnecessary queries, a download, and a re-upload of the data.

Describe alternatives you've considered

  • Make BigFrames.pandas.DataFrame less compatible with the pandas interface to avoid confusion by pandas-gbq.
  • Still download and re-upload but skip the schema detection, which should be unnecessary with a BigFrames DataFrame.

Additional context

See internal issue 374235961

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label Nov 5, 2024
@shollyman shollyman assigned chalmerlowe and Linchin and unassigned whuffman36 Nov 14, 2024
@Linchin Linchin added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Nov 26, 2024
@tswast tswast assigned tswast and unassigned chalmerlowe and Linchin Dec 10, 2024
@tswast
Copy link
Collaborator Author

tswast commented Dec 10, 2024

Since I'm trying to have a light day today due to OOO 1/2 day, I can pick this up, since I think I can get it done relatively quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
4 participants