Test cases written in moto makes actual AWS API calls to botocore instead of mocking them. This happens with the latest version of boto3 (1.8.). It used to work fine without issues with 1.7. versions.

Sample code to reproduce error

import boto3
import json
from moto import mock_s3

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

Expected result

Method should return the ListBuckets response. It should look something like:

{"Owner": {"DisplayName": "webfile", "ID": "bcaf1ffd86f41161ca5fb16fd081034f"}, "Buckets": [{"CreationDate": "2006-02-03 16:45:09+00:00", "Name": "testbucket"}], "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "HTTPHeaders": {"Content-Type": "text/plain"}}}

Actual error

botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Full stack trace

Traceback (most recent call last):
  File "testcases.py", line 14, in <module>
    test_mock_s3()
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "testcases.py", line 8, in test_mock_s3
    client.create_bucket(Bucket='testbucket')
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Library versions

moto : 1.3.4
boto3 : 1.8.1 - fails
boto3 : 1.7.84 - succeeds

Features

• support lambda messages from SNS
• run lambda in docker container
• support all lambda environments via docker
• fixes lambda region + arn
• adds partial cloudwatchlogs endpoint
• fixes environment variable propagation to lambda

Resolves

• https://github.com/spulec/moto/issues/1047
• https://github.com/spulec/moto/issues/981
• escaping issue with urls that contain backslashes

Just a placeholder till I actually start work later. If anyone wants to take over, just comment.

TODO:

• [x] CreateComputeEnvironment
• [x] DeleteComputeEnvironment
• [x] UpdateComputeEnvironment
• [x] DescribeComputeEnvironments
• [x] CreateJobQueue
• [x] DeleteJobQueue
• [x] UpdateJobQueue
• [x] DescribeJobQueues
• [x] RegisterJobDefinition
• [x] DeregisterJobDefinition
• [x] DescribeJobDefinitions
• [x] SubmitJob
• [x] CancelJob
• [x] TerminateJob
• [x] DescribeJobs
• [x] ListJobs

CloudFormation Todo:

• [x] ComputeEnvironment
• [x] JobQueue
• [x] JobDefinition

See https://stackoverflow.com/q/53220953/562769 / travis log :

I get the error message

botocore.exceptions.NoCredentialsError: Unable to locate credentials

Any idea why?

Environment variables

- AWS_ACCESS_KEY_ID=dummy-access-key
- AWS_SECRET_ACCESS_KEY=dummy-access-key-secret
- AWS_DEFAULT_REGION=us-east-1

Software Versions

• boto: 2.49.0
• boto3: 1.9.41
• botocore: 1.12.41
• moto: 1.3.7

Just a question really, but with Lambdas now executing within docker containers, is it possible for mocked lambdas to access other mocked resources, such as DynamoDB tables?

Also, do functioning environment variables get created when passed in to create_function?

Probably something @thehesiod can answer?

FYI @kbalk - this adds the env vars to the Lambda Docker container per your suggestion.

Marking it as draft for now, as the tests will probably fail..

This is a followup question to the PR "Feature: Custom cloudformation resources" and perhaps related to the issue "switch server mode to proxies".

The lambda submitted as a test case for that PR issued API calls to the EC2 service. Part of the fix required the lambda to specify the endpoint_url to any boto3 services it invoked, i.e.:

ec2 = boto3.client("ec2", endpoint_url="http://host.docker.internal:5000")

That lambda (or the test case) can be invoked in different environmentss:

• against AWS itself (no endpoint_url value needed),
• when LocalStack and moto are started up from the command line (the endpoint_url of "http://host.docker.internal:5000" works great here),
• when LocalStack and moto are started within a docker container and then start up the lambda within that container.

I have not been able to figure out the appropriate endpoint_url for the last case. I also don't have a way to determine which of the three situations I'm in. If I relied on LocalStack, I could use the environment variable LOCALSTACK_HOSTNAME to identify when I'm running against AWS versus localstack. However, if I only use moto endpoints, I don't have an equivalent environment variable.

Since my test case uses Terraform, I've tried passing variables declared in the Terraform modules, but I still can't distinguish between the three scenarios and I'm trying to avoid hardcoding the moto port.

Do you have any suggestions as to how I might solve this problem? I could live with hardcoding the moto port if I knew the appropriate endpoint for the third case above and if I had a way of knowing which endpoint I should be using.

At first look, I thought this was a duplicate of https://github.com/spulec/moto/issues/303.

However, on further inspection, the problem is only present with the mock_s3 decorator, not with the mock_s3_deprecated decorator. Therefore, if my inspection of the code base is right, this looks like a problem related to the responses library or the use of it in moto, whereas https://github.com/spulec/moto/issues/303 concerned the HTTPretty library.

Anyway, the meat of the issue:

After using @mock_s3 in django tests, other test classes that use LiveserverTestCase and the requests library stop working, and fail with the error: ConnectionError: Connection refused.

This is not an inherent problem with responses itself, as the following test demonstrates:

python -c "from moto.packages.responses.responses import mock; import requests; mock.start(); mock.stop(); print requests.get('http://google.com')"
<Response [200]>

However, if mock.stop() is omitted, you see the same error as in the django test:

python -c "from moto.packages.responses.responses import mock; import requests; mock.start(); print requests.get('http://google.com')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 302, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 244, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused: GET http://google.com/

This suggests to me that, one way or another, moto is failing to call mock.stop().

moto version 1.0.1 Django version 1.11.3

Versions

• boto==2.49.0
• boto3==1.9.225
• botocore==1.12.225
• moto==1.3.13

This issue has been raised many times before it appears, however after reading through all the reports and changing versions up and down to no avail, I've decided to raise another issue. When I try to mock out AWS using moto, I am getting the following error:

(InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

I've tried setting fake credentials using os.environ, I've tried deleting my credentials from the cli configuration, I've tried downgrading botocore/boto3 versions, I've tried re-arranging imports. None of this has helped unfortunately. Hopefully I'm missing something, but if not, there is something wrong.

The code

@mock_sqs
@mock_s3
@mock_sns
def test_bad_data(self):
    ...
    response = function.lambda_handler(
        None,
        None
    )
    ...

Calling the lambda_handler from the test is what lead us to the error:

def lambda_handler(event, context):
    ...
    try:
        ...
        s3 = boto3.resource('s3', region_name="eu-west-2")
        sqs = boto3.client('sqs', region_name="eu-west-2")
        lambda_client = boto3.client('lambda', region_name="eu-west-2")
        ...
        input_file = read_from_s3(s3, config['bucket_name'], config['file_name'])
        ...

It is where the read_from_s3 function is called that it tells me I have invalid credentials, which is the first instance of connecting to AWS. For reference, here is the read_from_s3 function:

def read_from_s3(s3, bucket_name, file_name):
    # Read from S3
    object = s3.Object(bucket_name, file_name)
    input_file = object.get()['Body'].read()

    return input_file

What was expected to happen Successfully mock AWS in tests.

What actually happens InvalidAccessKeyId error, suggesting that moto mocking isn't working.

If I've missed any important details please let me know so I can update.

Supported basic AWS IoT and IoTDataPlane operations. Error handling is not enough though.

skipping test of iot and iot-data when TEST_SERVER_MODE=true, bacause the process of authentication of aws-iot is complicated...

-----------------------
iot - 45% implemented
-----------------------
[ ] accept_certificate_transfer
[X] attach_principal_policy
[X] attach_thing_principal
[ ] cancel_certificate_transfer
[ ] create_certificate_from_csr
[X] create_keys_and_certificate
[X] create_policy
[ ] create_policy_version
[X] create_thing
[X] create_thing_type
[ ] create_topic_rule
[ ] delete_ca_certificate
[X] delete_certificate
[X] delete_policy
[ ] delete_policy_version
[ ] delete_registration_code
[X] delete_thing
[X] delete_thing_type
[ ] delete_topic_rule
[ ] deprecate_thing_type
[ ] describe_ca_certificate
[X] describe_certificate
[ ] describe_endpoint
[X] describe_thing
[X] describe_thing_type
[X] detach_principal_policy
[X] detach_thing_principal
[ ] disable_topic_rule
[ ] enable_topic_rule
[ ] get_logging_options
[X] get_policy
[ ] get_policy_version
[ ] get_registration_code
[ ] get_topic_rule
[ ] list_ca_certificates
[X] list_certificates
[ ] list_certificates_by_ca
[ ] list_outgoing_certificates
[X] list_policies
[X] list_policy_principals
[ ] list_policy_versions
[X] list_principal_policies
[X] list_principal_things
[X] list_thing_principals
[X] list_thing_types
[X] list_things
[ ] list_topic_rules
[ ] register_ca_certificate
[ ] register_certificate
[ ] reject_certificate_transfer
[ ] replace_topic_rule
[ ] set_default_policy_version
[ ] set_logging_options
[ ] transfer_certificate
[ ] update_ca_certificate
[X] update_certificate
[X] update_thing

-----------------------
iot-data - 75% implemented
-----------------------
[X] delete_thing_shadow
[X] get_thing_shadow
[ ] publish
[X] update_thing_shadow

Here is what I do:

mamba env create -n test-moto  # this creates a clean environment.
conda activate test-moto
pip install moto  # OR mamba install moto -c conda-forge
moto_server -p 5555

then in a different terminal I run

curl "http://localhost:5555/"

Expected behavior: The curl should immediately return a result.

Actual behavior: The moto_server hangs. The process is at 100% CPU usage so maybe it is busy waiting for something (just a guess).

If werkzeug is manually downgraded to v2.1.0 (pip install Werkzeug=2.1.0) or previous everything works as expected.

Here is my complete pip freeze:

-- edit by @bblommers - removed to clean the thread a bit

Currently there's no implementation of Describe Security Group Rules. I would love to take up this issue and implement it.

Here's my approch create an describe_security_group_rules method in the SecurityGroupBackend class which resides in moto/ec2/models/security_groups.py

However i'm a bit uncertain on the contents on describe_security_group_rules method. Not sure what to return from the method. Here is what i think

def describe_security_group_rules(self, filters=None):
    import pdb; pdb.set_trace()
    all_groups = self.groups.copy()
    matches = itertools.chain(*[x.copy().values() for x in all_groups.values()])
    if filters:
        matches = [grp for grp in matches if grp.matches_filters(filters)]
    return matches

This gets all the security groups that matches the filters. Now i'm not sure how the get the ingress/egress rules. Would be great if someone could enlighten me on this.

Also do i need to make any additional changes so that the describe_security_group_rules method gets fired when the aws ec2 describe-security-group-rules cmd is run

Dynamodb can only be mocked when credentials are taken from environment variables. This is too risky to be used in production(because we may have engineers who have production aws configs on their local computer -- and moto may not mock in case boto is instantiated beforehand). We need a feature where in the worst case scenario boto tries connecting to the local database.

I'll try to make a pull request myself. Please tell me if I got something wrong.

ps. we want to run moto in memory with pytest so that we may parallelize tests

Currently the secretsmanager component is missing implementation for a few endpoints, CancelRotateSecret is one of them. I use this library extensivley and just recently came accross the requirement for this feature, happy to take a crack and submit a PR.

Problem

It appears that the get_object_attributes on the s3 client does not return the expected XML for the object, instead simply the file contents. I noticed this while trying to work with different checksum algorithms but on further testing, the same issue presents itself even if ChecksumAlgorithm is not passed. An XML validation error is raised showing the xml_string in the parsing stage is the object body.

Code to reproduce

import boto3
import unittest
from moto import mock_s3

@mock_s3
class Test_Amazon(unittest.TestCase):

    def setUp(self):

        self.s3 = boto3.client('s3')
        self.s3.create_bucket(
            Bucket="bucket_name",
            CreateBucketConfiguration={"LocationConstraint":"eu-west-2"}
        )

    def test_digest_crc32(self):

        self.s3.put_object(
            Bucket="bucket_name",
            Key="file-1.txt",
            Body=b"Content",
            ChecksumAlgorithm="CRC32"
        )

        response = self.s3.get_object_attributes(
            Bucket='bucket_name',
            Key='file-1.txt',
            ObjectAttributes=['Checksum']
        )

EXPECTED

{'ResponseMetadata': {'RequestId': '0B5XW5F3....', 'HostId': '5h+IJazxYUjgZAxrqsfcsnjtiOTNU..., 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '5h+IJazxYUjgZAxrqsfcsnjtiOTNUd4h4LS....', 'x-amz-request-id': '0B5XW5F...', 'date': 'Thu, 29 Dec 2022 17:35:55 GMT', 'last-modified': 'Thu, 29 Dec 2022 15:09:13 GMT', 'server': 'AmazonS3', 'content-length': '244'}, 'RetryAttempts': 0}, 'LastModified': datetime.datetime(2022, 12, 29, 15, 9, 13, tzinfo=tzutc()), 'Checksum': {'ChecksumSHA256': 'R70pB1+LgBnwvuxthr7afJv2eq8FBT3L4LO8tjloUX8='}}

ACTUAL

self = <botocore.parsers.RestXMLParser object at 0x1151548b0>
xml_string = b'Content'

    def _parse_xml_string_to_dom(self, xml_string):
        try:
            parser = ETree.XMLParser(
                target=ETree.TreeBuilder(), encoding=self.DEFAULT_ENCODING
            )
            parser.feed(xml_string)
            root = parser.close()
        except XMLParseError as e:
>           raise ResponseParserError(
                "Unable to parse response (%s), "
                "invalid XML received. Further retries may succeed:\n%s"
                % (e, xml_string)
            )
E           botocore.parsers.ResponseParserError: Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
E           b'Content'

venv/lib/python3.8/site-packages/botocore/parsers.py:505: ResponseParserError
- generated xml file: /var/folders/85/z5lglbqs3_v1h9ldf1w1671m0000gn/T/tmp-879dlMlpwqeiX6w.xml -

Versions

Python 3.8.2 moto==4.0.11

Small "fix" to add in TargetKeyIds to canonical aliased kms keys.

These keys are a curiosity to me. I can't seem to find a way to initialize them via boto3 in actual AWS land. If you setup, say, ssm and add a first key to it using the service kms key it auto-generates. \o/.