Skip to content

feat: read_gbq suggests using BigQuery DataFrames with large results #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 20, 2024

Conversation

tswast
Copy link
Collaborator

@tswast tswast commented May 16, 2024

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes internal issue 340937138
🦕

@tswast tswast requested review from a team as code owners May 16, 2024 20:09
@tswast tswast requested a review from leahecole May 16, 2024 20:09
@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label May 16, 2024
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label May 16, 2024
@tswast
Copy link
Collaborator Author

tswast commented May 16, 2024

tests/system/test_read_gbq_with_bqstorage.py::test_large_results looks like a real failure. Investigating now.

Edit: Should be fixed by 6ba62d1

@@ -1050,12 +1088,7 @@ def to_gbq(
)

if api_method == "default":
# Avoid using parquet if pandas doesn't support lossless conversions to
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per coverage report, this was dead code. Our minimum pandas version is beyond the one where this feature wasn't available.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you this is a helpful comment!

Comment on lines +1 to +3
# Copyright (c) 2024 pandas-gbq Authors All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just double checking that this is normal given that we use a difference license header for other Google stuff

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not normal, but expected. pandas-gbq was originally split off from pandas itself, so we can't use the Google header. It's not 100% Google's copyright since not all contributions were under the CLA.

@@ -1050,12 +1088,7 @@ def to_gbq(
)

if api_method == "default":
# Avoid using parquet if pandas doesn't support lossless conversions to

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you this is a helpful comment!

@tswast tswast merged commit f937edf into main May 20, 2024
26 checks passed
@tswast tswast deleted the b340937138-plg-bigframes branch May 20, 2024 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants