I'm migrating my app from the depriciated oAuth2Client to this library. I use requests_oauthlib to get the refresh_token, the access_token and all the rest. Then I build the credential object like this:

    credentials = google.oauth2.credentials.Credentials(
                                                        access_token,
                                                        refresh_token = refresh_token,
                                                        token_uri = 'https://accounts.google.com/o/oauth2/token',
                                                        client_id = settings.GOOGLE_CLIENT_ID,
                                                        client_secret = settings.GOOGLE_CLIENT_SECRET
                                                        )

I'm not sure about the token_uri, what am I supposed to put there? In any case this seems to work, making a simple call like this one:

    authed_session = AuthorizedSession(credentials)
    response = authed_session.get('https://www.googleapis.com/oauth2/v1/userinfo') 

works as expected and I do get back the results. However my app relies a lot on the Google Analytic api, which uses its own build method.

With oAuth2client I used to do (to get properties using v3):

    http = httplib2.Http()
    http = credentials.authorize(http)
    v3 = build('analytics', 'v3', http=http)
    try:
        account_summaries = v3.management().accountSummaries().list().execute()
    except Exception as error:
        return error_page(request, error)
    google_email = account_summaries['username']

And to get metrics with v4:

    http = credentials.authorize(httplib2.Http())
    DISCOVERY_URI = ('https://analyticsreporting.googleapis.com/$discovery/rest')
    analytics = build('analytics', 'v4', http=http, discoveryServiceUrl=DISCOVERY_URI)

and then: analytics.reports().batchGet( ...........etc)

However now that I migrate to this library I don't know how I can use the build method. All the documentation is really old and still referencing the depreciated library. How can I authenticate and use correctly google-auth with the Google Analytics API?

Thanks

Original issue: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4358

After successful use of credentials, _ = google.auth.default(), an application crashes when credentials cannot be detected:

...
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 212, in __init__
    Client.__init__(self, credentials=credentials, _http=_http)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 125, in __init__
    credentials, _ = google.auth.default()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

/cc @dmho418

From @Spikey123 on August 23, 2017 13:5

Hello.

I've been using tswast's method to query BigQuery results and it's been purring along fine for a while now. Over the last several days, I've been getting an SSL certificate verification issue. Below is the full error message and below that is the code I use.

I'll note that I updated the python client, bigquery library, cloud storage library, and gcloud components. pip install --upgrade google-api-python-client pip install --upgrade google-cloud-bigquery pip install --upgrade google-cloud-storage gcloud components update

Service account permissions should be fine

Update: I'm getting the same exact error on other scripts I have running - like this one that deletes blobs in Google Cloud Storage:

Traceback (most recent call last):
  File "sfyExport/sfyExport.py", line 334, in <module>
    compression="GZIP" if args.gzip else "NONE")
  File "sfyExport/sfyExport.py", line 203, in main
    didDelete = delete_blobs_matching("mwg-bigqueryexport", deleteString)
  File "sfyExport/sfyExport.py", line 50, in delete_blobs_matching
    blobs = list_blobs(bucket_name)
  File "sfyExport/sfyExport.py", line 38, in list_blobs
    bucket = storage_client.get_bucket(bucket_name)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/client.py", line 173, in get_bucket
    bucket.reload(client=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/_helpers.py", line 99, in reload
    _target_object=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

I checked the certs on accounts.google.com and they appeared to be fine. So I'm not sure why the SSL error would be thrown.

Thanks!

Traceback (most recent call last):
  File "bqTests.py", line 295, in <module>
    args.num_retries)
  File "bqTests.py", line 254, in main
    failCount = async_query("SELECT Count(*) from [Tests.{0}]".format(bqTest["tempTable"]))
  File "bqTests.py", line 104, in async_query
    query_job.begin()
  File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 380, in begin
    method='POST', path=path, data=self._build_resource())
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

The code is:

87- # [START wait_for_job]
88- def wait_for_job(job):
89-     while True:
90-        job.reload()  # Refreshes the state via a GET request.
91-        if job.state == 'DONE':
92-            if job.error_result:
93-                raise RuntimeError(job.errors)
94-            return
95-        time.sleep(1)
96- # [END wait_for_job]

99- # [START async_query]
100- def async_query(query):
101-    client = bigquery.Client()
102-    query_job = client.run_async_query(str(uuid.uuid4()), query)
103-    query_job.use_legacy_sql = True
104-    query_job.begin()
105-
106-    wait_for_job(query_job)
107-
108-    rows = query_job.results().fetch_data(max_results=10)
109-    
110-    for row in rows:
111-        return row[0]
112-    
113- # [END async_query]

Copied from original issue: GoogleCloudPlatform/google-cloud-python#3857

Environment details

• OS: Linux, Mac
• Python version: 3.7, 3.8
• pip version: 20.3.3
• google-auth version: 1.22.1

Steps to reproduce

We found this bug while using dvc and using Google Cloud Storage as a backend. Authentication with google is done via a service key file. When running dvc with rsa==4.6 everything works fine, but when upgrading to rsa==4.7, we encounter the following error:

  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/dvc/tree/gs.py", line 139, in isfile
    return blob.exists()
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 484, in exists
    _target_object=None,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 431, in api_request
    timeout=timeout,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 289, in _make_request
    method, url, headers, data, target_object, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 327, in _do_request
    url=url, method=method, headers=headers, data=data, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 460, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 361, in refresh
    access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{"error":"invalid_grant","error_description":"Invalid JWT Signature."}')

Using the google.auth.default() helper to access credentials for the default service account on App Engine standard works great, but does not allow one to use auth scopes beyond the default scopes for the Python 3.7 and 3.8 runtimes. This is different to how the old 2.7 runtime works.

This app works on 2.7, fails with an exception on 3.8:

# main.py, requires flask, google-api-python-client
import flask
from googleapiclient import discovery

app = flask.Flask(__name__)
@app.route('/')
def home():
    service = discovery.build('drive', 'v3')

    request = service.files().list()
    response = request.execute()

    return response

On 3.8 the error is

googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v3/files?alt=json returned "Insufficient Permission: Request had insufficient authentication scopes.">" 

The problem is that google.auth.default() returns an instance of the Compute Engine credentials for Python 3, and that class does not allow changing scopes.

The credentials class uses the metadata service to get an access token. On App Engine standard the metadata service also allows one to request an access token with additional scopes (I haven't checked if this is also possible on Compute Engine or Flex). I have an implementation of ServiceAccountCredentials that supports requesting a token with additional scopes, which works on App Engine standard Python 3.

https://gist.github.com/davidwtbuxton/525924b7f06f56b8530947d55bad1c21

With that code, the service discovery can request the required scopes:

    credentials = ServiceAccountCredentials()
    service = discovery.build('drive', 'v3', credentials=credentials)

It would be cool if we could get this supported using google.auth.default() for Python 3 on App Engine standard. In particular it simplifies a lot of code that may mess around loading credentials from a JSON file or similar.

Would a PR for this feature be accepted?

Thanks,

David

Is there a way to disable the warning _CLOUD_SDK_CREDENTIALS_WARNING, maybe by setting an environment variable? We have an application where we want users to use their gcloud user credentials since it is run on their local machine. If we ran the app as a service on a server, then a service account would make more sense.

UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.

verify_id_token is properly splitting the jwt encoded string, but it isn't handling token components that aren't padded well.

verify_id_token should handle this and add padding as needed

EDIT: context, I'm following https://firebase.google.com/docs/auth/admin/verify-id-tokens.

I had to jump through a number of hoops to get the requests transport working in the App Engine environment:

1. Enable billing for the project
2. Vendor in the requests-toolbelt library and enable the monkeypatch (as documented here)
3. Add the ssl library to the app.yaml:

libraries:
- name: ssl
  version: latest

4. Enable sockets as documented here

Only by doing all the above, that I could get the HTTPS request to Google OAuth2 token servers working. Is this to be expected? Is there anything that can be done to make using this library in App Engine easier?

Function _unverified_decode assumes header is a dict and performs a .get("alg") after decoding the JWT token, however the header may be a different object which does not have the get method. Similar problem for the payload segment, which is accessed as a dictionary but there is no check for it.

This PR fixes the problem by checking the type of header/payload in the _unverified_decode function.

I'm attempting to update a G Suite user with the data in the following dictionary:

{
    "organizations": [
        {
            "customType": "",
            "location": "Tokyo, Japan",
            "name": "Example Inc.",
            "primary": true,
            "title": "FUD evangalist",
            "type": "work"
        }
    ]
}

This is the URL I'm using: URL = 'https://www.googleapis.com/admin/directory/v1/users/[email protected]'

Looking at the documentation, it looks to me like I should be able to PUT the dictionary directly like this: response = authed_session.put(URL, data = dictionary_to_send)

This gives me the error:

This API does not support parsing form-encoded input.

The I try to send the dictionary as JSON: response = authed_session.put(URL, data = json.dumps(dictionary_to_send))

This gives me the users data back, as expected. But, the user has not been updated with the data in the dictionary.

I have pasted the dictionary as a JSON in to the API Explorer. The changes are applied when I do that.

Any suggestions?

from_user_info currently always sets the token to None. So a credential created from a JSON is always invalid and must be refreshed.

>>> from google.oauth2 import credentials
>>> TOKENS = 'tokens.json' # OAuth2 token storage
>>> creds = credentials.Credentials.from_authorized_user_file(TOKENS)
>>> creds.valid
False
>>> creds.expiry
>>> creds.expired
False

https://github.com/googleapis/google-auth-library-python/blob/772dac6a6512230d32cb0dfae65a1a6aa9015049/google/oauth2/credentials.py#L235-L244

If a token is available, from_user_info should use it. info.get('token', None)

When we reach this line of code, we get "TypeError: unsupported type for timedelta seconds component: str"

https://github.com/googleapis/google-auth-library-python/blob/6c1297c4d69ba40a8b9392775c17411253fcd73b/google/oauth2/_client.py#L123

It would appear that the response from the token server has expires_in as a string:

Request:

https://securetoken.googleapis.com/v1/token?key=AI... 
{
  'grant_type': 
  'refresh_token', 
  'client_id': '76...', 
  'client_secret': '...', 
  'refresh_token': 'AO...'
}

Response:

{
  'access_token': 'ey...', 
  **'expires_in': '3600'**, 
  'token_type': 'Bearer', 
  'refresh_token': 'AO...', 
  'id_token': 'ey...', 
  'user_id': 'Xs...', 
  'project_id': '23...'
}

Seems this code should expect a string and not an integer.

This short chain demonstrates a missed expectation, that the google-auth package will mask the presence of other packages in a google namespace when installed. This issue affects corporate packages internally.

The pkg_resources style namespace is the least preferable and least compatible technique indicated in the docs.

By removing the pkg_resources technique, it makes the pkgutil technique the default, adds compatibility with native namespace packages.

After Python 2.7 support is dropped, this package could consider dropping the pkgutil technique as well and instead use native namespace packages, but that's a consideration for a later time.

Environment details

• Python version: 3.10
• google-auth version: 2.14.1

Steps to reproduce

We use the google-storage-python library to upload some files to GCS and, occasionally, we get errors from the google-auth library. Sometimes the run would work, sometimes a single upload would fail, sometimes multiple uploads would fail.

I've spent some time looking into the code and, while it looks like retries have been implemented in this PR ( https://github.com/googleapis/google-auth-library-python/pull/1113 ), I am not sure if the behavior also applies here ( https://github.com/googleapis/google-auth-library-python/blob/main/google/auth/transport/requests.py#L414-L418 ).

Why not use the default Retry object defined in the core-api? https://github.com/googleapis/python-storage/blob/3e4ea84163a6160703d219bea99e46b1e3965722/google/cloud/storage/retry.py#L57

2022-12-13T13:26:11.8192323Z 2022-12-13 13:25:53,991 - distributed.worker - WARNING - Compute Failed
2022-12-13T13:26:11.8192894Z Key:       upload_directory_to_gcs-52fd83e3-65a1-4bb8-890a-09f0b7d57d1d
2022-12-13T13:26:11.8193294Z Function:  execute_task
2022-12-13T13:26:11.8198326Z args:      ((<function apply at 0x7fc44c6df520>, <function upload_directory_to_gcs at 0x7fc41a282a70>, ['/vsigs/overstory-customer-test/risk_score_tests/dev_dagster_risk_score/no_chm/gd0a49e1c-1dec-4993-9193-8982afc7adce/', '/tmp/tmpzwg8ey_m/risk_score_span_38424'], (<class 'dict'>, [['delete_dir_after_cp', True]])))
2022-12-13T13:26:11.8199004Z kwargs:    {}
2022-12-13T13:26:11.8199650Z Exception: 'ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host=\'storage.googleapis.com\', port=443): Read timed out. (read timeout=60)"))'
2022-12-13T13:26:11.8200071Z 
2022-12-13T13:26:11.8230753Z 2022-12-13 13:25:54 +0000 - dagster - ERROR - encroachment_risk_score_graph - ff97359e-2a45-43e5-b2ed-221a90a7e208 - 4534 - risk_score_op - STEP_FAILURE - Execution of step "risk_score_op" failed.
2022-12-13T13:26:11.8231250Z 
2022-12-13T13:26:11.8235148Z dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "risk_score_op"::
2022-12-13T13:26:11.8235581Z 
2022-12-13T13:26:11.8235813Z requests.exceptions.ReadTimeout: None: None
2022-12-13T13:26:11.8236024Z 
2022-12-13T13:26:11.8236128Z Stack Trace:
2022-12-13T13:26:11.8244030Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
2022-12-13T13:26:11.8244686Z     yield
2022-12-13T13:26:11.8245317Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 421, in iterate_with_context
2022-12-13T13:26:11.8245831Z     next_output = next(iterator)
2022-12-13T13:26:11.8275759Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 73, in _coerce_solid_compute_fn_to_iterator
2022-12-13T13:26:11.8277774Z     result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
2022-12-13T13:26:11.8278419Z   File "/home/runner/work/risk_score/risk_score/risk_score/dagster/ops/span_level_encroachment.py", line 208, in risk_score_op
2022-12-13T13:26:11.8278917Z     return _run_dask_span_level_encroachment(
2022-12-13T13:26:11.8279482Z   File "/home/runner/work/risk_score/risk_score/risk_score/dask_runner.py", line 582, in _run_dask_span_level_encroachment
2022-12-13T13:26:11.8279947Z     run_dask_risk_score_by_span(
2022-12-13T13:26:11.8280461Z   File "/home/runner/work/risk_score/risk_score/risk_score/dask_runner.py", line 444, in run_dask_risk_score_by_span
2022-12-13T13:26:11.8280927Z     dask_client.gather(futures)
2022-12-13T13:26:11.8282513Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/distributed/client.py", line 2291, in gather
2022-12-13T13:26:11.8283044Z     return self.sync(
2022-12-13T13:26:11.8283619Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 339, in sync
2022-12-13T13:26:11.8283986Z     return sync(
2022-12-13T13:26:11.8284549Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 406, in sync
2022-12-13T13:26:11.8285009Z     raise exc.with_traceback(tb)
2022-12-13T13:26:11.8285586Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 379, in f
2022-12-13T13:26:11.8286199Z     result = yield future
2022-12-13T13:26:11.8286824Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
2022-12-13T13:26:11.8287262Z     value = future.result()
2022-12-13T13:26:11.8289132Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/distributed/client.py", line 2154, in _gather
2022-12-13T13:26:11.8289705Z     raise exception.with_traceback(traceback)
2022-12-13T13:26:11.8290317Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/dask/utils.py", line 71, in apply
2022-12-13T13:26:11.8291635Z     return func(*args, **kwargs)
2022-12-13T13:26:11.8292200Z   File "/home/runner/work/risk_score/risk_score/risk_score/risk_score_runner.py", line 50, in upload_directory_to_gcs
2022-12-13T13:26:11.8292784Z     upload_to_gcs(
2022-12-13T13:26:11.8293380Z   File "/home/runner/work/risk_score/risk_score/risk_score/risk_score_runner.py", line 102, in upload_to_gcs
2022-12-13T13:26:11.8294024Z     blob.upload_from_filename(local_path)
2022-12-13T13:26:11.8294850Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2683, in upload_from_filename
2022-12-13T13:26:11.8295409Z     self.upload_from_file(
2022-12-13T13:26:11.8296187Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2539, in upload_from_file
2022-12-13T13:26:11.8296801Z     created_json = self._do_upload(
2022-12-13T13:26:11.8297494Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2354, in _do_upload
2022-12-13T13:26:11.8298136Z     response = self._do_multipart_upload(
2022-12-13T13:26:11.8298886Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 1889, in _do_multipart_upload
2022-12-13T13:26:11.8299493Z     response = upload.transmit(
2022-12-13T13:26:11.8300240Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 153, in transmit
2022-12-13T13:26:11.8300884Z     return _request_helpers.wait_and_retry(
2022-12-13T13:26:11.8301691Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 171, in wait_and_retry
2022-12-13T13:26:11.8312341Z     raise error
2022-12-13T13:26:11.8313401Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 148, in wait_and_retry
2022-12-13T13:26:11.8366773Z     response = func()
2022-12-13T13:26:11.8367844Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 145, in retriable_request
2022-12-13T13:26:11.8368516Z     result = transport.request(
2022-12-13T13:26:11.8369254Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/google/auth/transport/requests.py", line 549, in request
2022-12-13T13:26:11.8369946Z     response = super(AuthorizedSession, self).request(
2022-12-13T13:26:11.8370703Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
2022-12-13T13:26:11.8372332Z     resp = self.send(prep, **send_kwargs)
2022-12-13T13:26:11.8373186Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
2022-12-13T13:26:11.8374620Z     r = adapter.send(request, **kwargs)
2022-12-13T13:26:11.8376155Z   File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/requests/adapters.py", line 578, in send
2022-12-13T13:26:11.8470574Z     raise ReadTimeout(e, request=request)

Any suggestion is welcome

:robot: I have created a release beep boop

2.16.0 (2022-12-14)

Features

• AwsCredentials should not call metadata server if security creds and region are retrievable through the environment variables (#1195) (5e27c8f)
• Wrap all python built-in exceptions into library excpetions (#1191) (a83af39)

This PR was generated with Release Please. See documentation.

I consider myself pretty familiar with the various google auth flows available via the python SDK - and how this interacts with gcloud-generated credentials.

We are using the bq SDK in the typical way; client = bigquery.Client() and we make use of ADC so our code is interoperable between dev and prod. Our code interacts with external tables that are sourced from sheets on google drive. We know that we need to provide the necessary scopes (and of course, permission to the underlying sheets).

The following works fine for a user identity with the necessary permissions: gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform

However, the following does not: gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform --impersonate-service-account=hand-of-god@mlg-apollo-data-prod.iam.gserviceaccount.com

We receive google.api_core.exceptions.Forbidden: 403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.

I can replicate the same issue with my user credential if 1 of the following 2 things are true:

1. I don't pass google drive scopes.
2. I don't have access to the underlying file.

The service account that I am impersonating definitely has access to the file and I can see the BigQuery job failure with non-descript error message (a feature request has been raised for this with the BigQuery REST API team). My suspicion is that when impersonating a service account, the scopes (that are presumably buried in the credential) are not passed through / correctly read by the SDK (WHEN the ADC was generated using SA impersonation only). Maybe a similar issue is happening with my above note when the project cannot be inferred from the environment.

See below screenshot proof of correct permissions being in place:

Very grateful for some direction here...