Been seeing the following issue in our flows.
Hypothesis I have explored.
- Doesn't seem to be any permission or improper setup thing, since dozens of identical branches triggered through foreach process just fine. And they have the exact s3 roles and permissions, so not even sure what the credentials could not be located message indicates.
- it executes every line of code from the step and does as expected, but then it leads to this timeout error during container exit at the end which makes the entire flow go haywire and fail.
- I thought maybe it has something to do with trying to snapshot large dataframes, but this error seems to appear randomly regardless of the amount of data being processed in the task.
Any insights?
timestamp | message
-- | --
1.6107E+12 | 2021-01-15 14:47:29.067 [40293/process_a_provider/211754 (pid 17969)] Task is starting.
1.6107E+12 | 2021-01-15 14:47:35.168 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status SUBMITTED)...
1.6107E+12 | 2021-01-15 14:48:01.009 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:48:01.009 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:48:31.192 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:49:01.335 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:49:31.467 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:50:01.628 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:50:33.037 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
1.6107E+12 | 2021-01-15 14:50:35.746 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status STARTING)...
1.6107E+12 | 2021-01-15 14:51:05.856 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status STARTING)...
1.6107E+12 | 2021-01-15 14:51:29.848 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNING)...
1.6107E+12 | 2021-01-15 14:51:39.622 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Setting up task environment.
1.6107E+12 | 2021-01-15 14:59:17.185 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /simple/awscli/
1.6107E+12 | 2021-01-15 15:03:31.869 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Downloading code package.
1.6107E+12 | 2021-01-15 15:03:41.894 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Code package downloaded.
1.6107E+12 | 2021-01-15 15:03:47.287 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Bootstrapping environment.
1.6107E+12 | 2021-01-15 15:08:27.341 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Environment bootstrapped.
1.6107E+12 | 2021-01-15 15:09:32.012 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting.
1.6107E+12 | 2021-01-15 15:10:31.520 [40293/process_a_provider/211754 (pid 17969)] [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Provider processed successfully - 0655011
1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid 17969)] Batch error:
1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid 17969)] Task crashed due to CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s .This could be a transient error. Use @retry to retry.
1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid 17969)]
1.6107E+12 | 2021-01-15 15:11:06.448 [40293/process_a_provider/211754 (pid 17969)] Task failed.
1.6107E+12 | 2021-01-15 15:11:07.060 [40293/process_a_provider/211754 (pid 18823)] Task fallback is starting to handle the failure.
1.6107E+12 | 2021-01-15 15:11:07.775 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 7 more times..
1.6107E+12 | 2021-01-15 15:11:09.790 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 6 more times..
1.6107E+12 | 2021-01-15 15:11:16.810 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 5 more times..
1.6107E+12 | 2021-01-15 15:11:20.825 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 4 more times..
1.6107E+12 | 2021-01-15 15:11:32.850 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 3 more times..
1.6107E+12 | 2021-01-15 15:11:50.880 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 2 more times..
1.6107E+12 | 2021-01-15 15:12:27.929 [40293/process_a_provider/211754 (pid 18823)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 1 more times..
1.6107E+12 | 2021-01-15 15:13:31.995 [40293/process_a_provider/211754 (pid 18823)] Internal error
1.6107E+12 | 2021-01-15 15:13:31.999 [40293/process_a_provider/211754 (pid 18823)] Traceback (most recent call last):
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/cli.py", line 853, in main
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] start(auto_envvar_prefix='METAFLOW', obj=state)
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py", line 764, in __call__
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] return self.main(args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py", line 717, in main
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] rv = self.invoke(ctx)
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] return _process_result(sub_ctx.command.invoke(sub_ctx))
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py", line 956, in invoke
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] return ctx.invoke(self.callback, ctx.params)
1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py", line 555, in invoke
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] return callback(args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] return f(get_current_context().obj, args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/cli.py", line 430, in step
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] max_user_code_retries)
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/task.py", line 278, in run_step
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] monitor=self.monitor)
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 34, in __init__
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] super(S3DataStore, self).__init__(args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/datastore.py", line 362, in __init__
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] self.save_metadata('attempt', {'time': time.time()})
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/datastore.py", line 49, in method
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] return f(self, args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 168, in save_metadata
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] self._put_s3_object(path, json.dumps(data).encode('utf-8'))
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/util/s3util.py", line 37, in retry_wrapper
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] raise last_exc
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/util/s3util.py", line 21, in retry_wrapper
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] return f(self, args, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 65, in _put_s3_object
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] self.s3.upload_fileobj(buf, url.netloc, url.path.lstrip('/'))
1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] return future.result()
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] return self._coordinator.result()
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] raise self._exception
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] return self._execute_main(kwargs)
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] return_value = self._main(kwargs)
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)] client.put_object(Bucket=bucket, Key=key, Body=body, extra_args)
1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)] return self._make_api_call(operation_name, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] operation_model, request_dict, request_context)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] return self._endpoint.make_request(operation_model, request_dict)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] return self._send_request(request_dict, operation_model)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] request = self.create_request(request_dict, operation_model)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] operation_name=operation_model.name)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] return self._emitter.emit(aliased_event_name, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] return self._emit(event_name, kwargs)
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] response = handler(kwargs)
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] return self.sign(operation_name, request)
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/signers.py", line 157, in sign
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] auth.add_auth(request)
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/auth.py", line 425, in add_auth
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] super(S3SigV4Auth, self).add_auth(request)
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] File "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] raise NoCredentialsError
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)] botocore.exceptions.NoCredentialsError: Unable to locate credentials
1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]
1.6107E+12 | 2021-01-15 15:13:32.477 [40293/process_a_provider/211754 (pid 18823)] Task failed.
1.6107E+12 | 2021-01-15 15:13:33.071 [40293/process_a_provider/211754 (pid 18937)] Task fallback is starting to handle the failure.
1.6107E+12 | 2021-01-15 15:13:33.763 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 7 more times..
1.6107E+12 | 2021-01-15 15:13:39.179 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 6 more times..
1.6107E+12 | 2021-01-15 15:13:42.798 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 5 more times..
1.6107E+12 | 2021-01-15 15:13:49.822 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 4 more times..
1.6107E+12 | 2021-01-15 15:14:00.841 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 3 more times..
1.6107E+12 | 2021-01-15 15:14:18.866 [40293/process_a_provider/211754 (pid 18937)] S3 datastore operation _put_s3_object failed (Unable to locate credentials). Retrying 2 more times..