Skip to content

BUG: Add support to replace partitions in date-partitioned tables (#43) #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

mremes
Copy link
Contributor

@mremes mremes commented Feb 19, 2018

Relates to issue #43

Changes:

  • Ability to write into a partition of a date-partitioned table by the module inferring insert into a partitioned table from the table decorator or if specified by user
  • Styling and docstring edits
  • Fix use schema when appending data to a table with an existing schema (ie. regular appends or date-partitioned tables inserts)
    Needs:
  • Tests for:
    • stuff is in the right partition
    • only the target partition is affected
    • ?

@codecov-io
Copy link

codecov-io commented Feb 19, 2018

Codecov Report

Merging #124 into master will decrease coverage by 44.74%.
The diff coverage is 25%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master     #124       +/-   ##
===========================================
- Coverage    75.7%   30.96%   -44.75%     
===========================================
  Files           8        8               
  Lines        1626     1634        +8     
===========================================
- Hits         1231      506      -725     
- Misses        395     1128      +733
Impacted Files Coverage Δ
pandas_gbq/gbq.py 20.64% <25%> (-55.73%) ⬇️
pandas_gbq/tests/test_gbq.py 26.88% <0%> (-56.76%) ⬇️
pandas_gbq/_load.py 62.5% <0%> (-35%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99cd849...4e0d557. Read the comment docs.

@mremes mremes force-pushed the dpt-support branch 3 times, most recently from 19d0d27 to 3a4f4a7 Compare February 19, 2018 12:49
@mremes mremes changed the title BUG: Add support to replace partitions in date-partitioned tables (#47) BUG: Add support to replace partitions in date-partitioned tables (#43) Feb 19, 2018
@tswast tswast self-requested a review February 20, 2018 00:54
@@ -699,8 +695,9 @@ def delete_and_recreate_table(self, dataset_id, table_id, table_schema):
table = _Table(self.project_id, dataset_id,
private_key=self.private_key)
table.delete(table_id)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that BigQuery lets you delete just a single partition using the table delete operation? I haven't tried that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it with the decorated table name and it worked as intended.

return True
except NotFound:
return False
except self.http_error as ex:
self.process_http_error(ex)

def create(self, table_id, schema):
def create(self, table_id, schema, date_partitioned=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than pass in date_partitioned here, I'd like to see an argument similar to the configuration argument in read_gbq which can take a dictionary in the JSON API format. That way options like timePartitioning can be set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense to me! I'll look into including the configuration in the to_gbq call to enable this and rest of the API configuration.

@tswast
Copy link
Collaborator

tswast commented Sep 24, 2020

Closing as pandas-gbq has changed quite a bit since this PR was opened.

@tswast tswast closed this Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants