ENH: Function to_gbq(): Dissociate the project ID to run the BQ job from the table project

[vreyespue]

I think it would be very helpful in certain situations to differentiate between the project where the BQ job is to be run, and the project in which the the table is located (especially because of the costs billing).
As far as I understood, at present this is not possible when calling the function to_gbq (link), because it gets only one project_id parameter as input, and the destination_table is in the form dataset.tablename.
For instance, in the python-bigquery API from Google, you can do this by using the method load_table_from_dataframe (link) which receives as input a TableReference object (link) (with a project_id, a dataset_id and a table_id), being however called from a client running jobs in a separate project_id.
I think this enhancement would be very useful in situations where splitting the costs among projects is important, while maintaining the availability of the data among projects and/or departments (which is the case in certain organizations).
Many thanks in advance for taking into consideration this issue, and please excuse me if I misunderstood something about the functionality of this method.
Best regards.

[ShantanuKumar]

You can actually even do it right now by attaching the project_id in front of the destination_table.
So you should pass project_id.dataset.tablename as the destination_table. Your project_id parameter for to_gbq can be the project where you want the billing.

[vreyespue]

Hi @ShantanuKumar, many thanks for your suggestion. I actually tried this, but got an error because the destination table must be exactly in the form dataset.tablename (i.e. two points are not allowed).

[ShantanuKumar]

@vreyespue

You need to use `` around the name so pass something like

destination_table = "`project_id.dataset.tablename` "

[vreyespue]

Hi @ShantanuKumar, many thanks for your comment.
I tried exactly what you suggest, i.e.

df_goal_table_typed.to_gbq(destination_table= "`project_id_data.dataset.tablename`",
                           project_id= "project_id_job")

And got this error message:

Traceback (most recent call last):
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 1312, in exists
    self.client.get_table(table_ref)
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 679, in get_table
    retry, method="GET", path=table_ref.path, timeout=timeout
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 556, in _call_api
    return call()
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/google/cloud/_http.py", line 421, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 GET https://bigquery.googleapis.com/bigquery/v2/projects/rd-bdlab-9c32/datasets/%60project_id.dataset/tables/tablename%60: Invalid dataset ID "`project_id.dataset". Dataset IDs must be alphanumeric (plus underscores and dashes) and must be at most 1024 characters long.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sc_2_classifier_cuuid.py", line 264, in <module>
    if_exists='append')
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas/core/frame.py", line 1546, in to_gbq
    credentials=credentials,
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas/io/gbq.py", line 219, in to_gbq
    private_key=private_key,
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 1202, in to_gbq
    if table.exists(table_id):
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 1317, in exists
    self.process_http_error(ex)
  File "/Users/vicente.reyes/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 423, in process_http_error
    raise GenericGBQException("Reason: {0}".format(ex))
pandas_gbq.gbq.GenericGBQException: Reason: 400 GET https://bigquery.googleapis.com/bigquery/v2/projects/rd-bdlab-9c32/datasets/%60project_id.dataset/tables/tablename%60: Invalid dataset ID "`project_id.dataset". Dataset IDs must be alphanumeric (plus underscores and dashes) and must be at most 1024 characters long.

Thus, it seems that project_id.dataset is interpreted as the dataset ID, which is obviously not correct.
Here my pandas-gbq version:

$ pip freeze | grep bq
pandas-gbq==0.13.0

Please let me know if you need further information.
Many thanks and best regards.

[ShantanuKumar]

I think you are right. It only works when running query using read_gbq. But in that case the table name is part of the query.
Something like

select count(*) from `project_id_data.dataset.tablename`

It doesn't work when you try to write to a table using to_gbq.

[vreyespue]

Dear @ShantanuKumar, thank you very much for your nice clarification.
I think that fits very well with the title and description of this issue, i.e. it doesn't work when calling the function to_gbq.
Many thanks and best regards 😄 👍

[vreyespue]

Dear @tswast is there any update on this issue? In my personal opinion, having open issues without any progress in ~5 months does not make too much sense. I would proceed to close this issue if there is no progress in the next few weeks. Many thanks for your understanding. Best regards.

[tswast]

We are open to pull requests to add a project ID parameter to to_gbq.

[vreyespue]

Dear @tswast, many thanks for your answer. I will take a look to see if I can implement it myself. Could you maybe suggest how would you approach this task? In other words, which files should be changed in order to implement this fix? Should I take something important into consideration? Many thanks and best regards.

[tswast]

This is the problematic line:

https://github.com/pydata/pandas-gbq/blob/ac2d2fe4ac0025109f8df3723e3f03a337face94/pandas_gbq/load.py#L59

It needs to be changed to accept a project ID. There will need to be some refactoring to add a project ID parameter to this function.

I recommend using TableReference.from_string, where default_project=client.project

[vreyespue]

Dear @tswast, cool many thanks for your advice. I will take a closer look and see what I can do 😄 best regards 👍