By testing and from user guide documentation (/blob/main/docs/user-guide/Examples.md), I find that RESTler compiler does not use examples in the swagger to construct a valid string for a number of properties. For example for `lastDoneBefore' in the swagger file we have:

- in: query
  name: lastDoneBefore
  description: Last done service before date and time or duration.
  examples:
    FullUtcTimestamp:
      value: 2008-01-23T12:05:01.700Z
    DateOnly:
      value: 2008-01-23T00:06:00.000Z
    Duration:
      value: '-P1DT8H5M10S'
      summary: >-
        1 day 8 hours 5 minutes and 10 seconds ago, value is
        case-sensitive
    DaysOnly:
      value: P3D
      summary: 3 day from now
    HoursOnly:
      value: '-PT8H'
      summary: 8 hours ago
    MinutesOnly:
      value: '-PT32M'
      summary: 32minutes ago
    SecondsOnly:
      value: PT100S
      summary: 100 seconds from now
  required: false
  schema:
    type: string

RESTler cannot interpret the examples in the schema above to construct a valid string for lastDoneBefore. This probably is the main cause for most of our 40x errors.

2021-02-05 04:20:29.127: Sending: 'GET /url/api/work?pageFromKey=fuzzstring&pageSize=0&pageTotal=true&sort=["fuzzstring"]&healthStatus=["good"]&type=["machine"]&tamperProtectionEnabled=true&lockdownStatus=["creatingWhitelist"]&lastDoneBefore=fuzzstring&lastDoneAfter=fuzzstring&ids=["fuzzstring"]&fields=["fuzzstring"]&view=basic HTTP/1.1\r\nAccept: */*\r\nHost: api-rg01.xyz.org\r\nContent-Type: application/json\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate, br\r\nX-KLM-ID: <...>\r\n_OMITTED_AUTH_TOKEN_\r\nContent-Length: 0\r\nUser-Agent: restler/7.2.0\r\n\r\n'

2021-02-05 04:20:29.411: Received: 'HTTP/1.1 400 \r\nDate: Tue, 05 Feb 2021 04:20:29 GMT\r\nContent-Type: application/json\r\nContent-Length: 231\r\nConnection: keep-alive\r\nX-Correlation-ID: <...>\r\nX-Request-ID: <...>\r\n\r\n{\n    "error": "illegalArgument",\n    "correlationId": "<...>",\n    "requestId": "<...>",\n    "createdAt": "2021-02-05T04:20:29.262Z",\n    "message": "Invalid request"\n}

Are there going to be any updates to RESTler's compiler for this matter?

Otherwise, do I have to add the property to fix custom strings using the "restler_custom_payload" field in dict.json? Is there any better solution, for example, adding to examples.json file?

Does this tool support Mac/Linux?

When I compiled the tool and ran the quick-start.py I saw in the error:

• python3 restler-quick-start.py --api_spec_path $REDACTED/data/file.json --restler_drop_dir $REDACTED/restler-fuzzer-7.2.0/restler_bin
• System.Exception: Platform not supported

The documentation only references Windows, so it would be nice to know if windows is the only intended target.

Hi! I defined in my (custom) dict.json that a fuzzable datetime object should look like this:

  "restler_fuzzable_datetime": [
    "2019-06-26T20:20:39+00:00"
  ]

However, when I run restler test, I see in the network testing log that It's testing with a group like this:

+ restler_fuzzable_datetime: ['"2019-06-26T20:20:39+00:00"', '"2021-10-17"', '"2021-09-15"']
                                                              ^---------------------------^
                                                                definitely not mine 

This leads to the tests failing since the format is not supported. How do I make sure only my dict is used?

... or what would be the best practice to achieve something similar, if custom payloads are not intended for this purpose? I am testing Restler on a very simple CRUD Api for TodoItems (id: Integer, name: String, isComplete: Boolean) and have provided the following custom payload in the dictionary:

"restler_custom_payload": {
  "name": [
    "fuzzstring", "ultrasuperduperannoyinglylongstringwithwhichtofuzz"
  ]
},

and introduced a very simple bug which I was hoping to "catch" with Restler. Trying to POST or PUT any TodoItem with a name longer than 20 characters returns 501. After 5 minutes in fuzz mode, Restler only uses the shorter string for any and all requests and thus does not discover the bug. The one exception and only reason I know the custom payload was correctly compiled is that Restler used the second, longer string once, in a sequence where it POSTed an item with the short name, deleted it and then tried to PUT an item with the long name at the same id. Resulting in a 404 because the item has been deleted.

If I reverse the order in this list, it will try to POST with the long name, get the 501, create a bug bucket, try the short string and (sensibly) use the short name in all further tests. I would like to know how I can test each value at least once, without needing to manually order them such that the correct ones come last and whether a custom payload is even the right way to do that.

Thanks, Joris

Hi, I have used latest image from Docker hub to setup Restler. Howerver, if I run test command, it does not send any requests. All request related files (runSummary.json, errorBuckets.json etc..) are empty. Following entries can be found in the logs: 2021-05-05T10:40:47.0744743Z Request coverage (successful / total): 0 / 1 2021-05-05T10:40:47.0745209Z No bugs were found. 2021-05-05T10:40:47.0745516Z Task Test succeeded. Am I doing anything wrong here?

This change adds a new option to track parameter combinations to the engine.

When the fuzzing mode is 'test-all-combinations', a new property 'tracked_parameters' will appear in speccov.json. This property contains key-value pairs of all of the parameters for which more than one combination is being tested.

In this commit, the parameters for which tracking is supported are:

• enums
• custom payloads

For example, an enum 'per_page' with several values will appear as:

    "tracked_parameters": {
        "per_page_9": "2"
    }

The suffix '9' is used to disambiguate between several primitives of the same name appearing in the payload (for example "location" appears several times in a body), and is the position of the argument in the request definition.

Full support for tracking fuzzable primitives will be enabled in a future update. For fuzzable primitives, the name of the parameter or property in the payload is not yet included, and a default 'tracked_param' name is used instead.

For example:

    "tracked_parameters": {
        "tracked_param_14": "1"
    }

Dear developers

I'm aware that this checker is currently off by default, but nevertheless I played around with it and have a few questions about it. It's basically three questions in one issue, but they are related to each other so I'm including them in one issue.

To test, I'm using the following REST API:

POST /rest/admin/products/{username} => This creates a product provided by user {username}. This can only be done by a user with a valid JSON Web Token (JWT) and only if the username in the token corresponds to {username}. It returns, among other data, the productID of the newly created product.

GET /rest/admin/products/{username}/{productID} => Allows to get a product. Can only be done with valid JWT that contains the corresponding username. So just like the POST request above, this is part of the "administrative area" of the API (anything below /rest/admin) to maintain products.

DELETE /rest/admin/products/{username}/{productID} => Allows to delete a product. Can only be done with valid JWT that contains the corresponding username. Also part of the "administrative area".

GET /rest/products/{productID} => Allows any user (no authentication required) to get a product. So in contrast to the requests above, this is not part of the "administrative area" of the API used for maintaining product, but allows anonymous users to view product details.

There's a little python program that returns valid JWTs of two users, luke and snoopy. This is used in parameter --token_refresh_command.

#!/usr/bin/env python
import sys
sys.stdout.write("{'luke':{}, 'snoopy':{}}\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJNYXJrZXRwbGFjZSIsInN1YiI6Imx1a2UiLCJleHAiOjE5MDczNTM5NzJ9.jdoYfz_diOQa...\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJNYXJrZXRwbGFjZSIsInN1YiI6InNub29weSIsImV4cCI6MTkwNzM1NDAxMX0.XiOtGu...\n")

I've added two vulnerabilities:

• With a valid JWT, one can GET the products of another user, e.g., snoopy can successfully execute GET /rest/admin/products/luke/25 to get a product that belongs to luke.
• With a valid JWT, one can DELETE the products of another user, e.g., snoopy can successfully execute DELETE /rest/admin/products/luke/25 to delete a product that belongs to luke.

The results of fuzz-lean are as follows:

• GET /rest/admin/products/luke/25 as user snoopy is correctly recognised as a vulnerability.
• GET /rest/products/25 is wrongly flagged as a vulnerability. Intuitively, I wouldn't have expected this as the POST request created the product below /rest/admin/products/luke. But apparently, a vulnerability is flagged if the corresponding productID is used in any other request, no matter what the full pat is. Is this intentional?
• DELETE /rest/admin/products/luke/25 as user snoopy is not recognised as a vulnerability. Apparently, when testing this, restler first does a POST and a corresponding DELETE (so the product is removed), and then it does another DELETE (I assume with the second user). This results in a 4xx error, as the product is longer there. This seems to be a wrong verification approach and a possible bug. The correct approach would be to do POST with the JWT of user1 and then DELETE with the JWT of user2.

And a final question: Is there a way to link the username in a token (that is returned by the python script above in my case) with the username that is used in the path of the POST request? Right now, I have hardcoded this in Grammar.py which works (so it always uses username luke), but this is not really an elegant solution.

So to summarise: three questions/issues:

• Is allowed access to GET /rest/products/25 intentional?
• Is the missed vulnerability with DELETE /rest/admin/products/luke/25 due to a bug?
• Is there a good way to link the username in a JWT to parameters used in requests?

For reference, I've attached Grammar.py, the network log and the two reported vulnerabilities.

Thanks a lot! Marc grammar.py.zip network.testing.140572845729536.1.txt NameSpaceRuleChecker_20x_1.txt NameSpaceRuleChecker_20x_2.txt

Hello,

I am having trouble getting past the Testing step as I am receiving 0% request coverage. On every endpoint request there is a 301 "Moved Permanently" error. I changed the API spec to contain only one endpoint so it is easier to work with / troubleshoot.

Here is the command I am running for test:

$ dotnet ~/restler-fuzzer/restler_bin/restler/Restler.dll test 
      --grammar_file Compile/grammar.py 
      --dictionary_file Compile/dict.json 
      --settings Compile/engine_settings.json 
      --token_refresh_command "python3 ~/restler-fuzzer/demo/token.py" 
      --token_refresh_interval 720 
      --no_ssl

Starting task Test...
Using python: 'python3' (Python 3.9.7)
Request coverage (successful / total): 0 / 1
No bugs were found.
Task Test succeeded.
Collecting logs...

Here is whats returned in the speccov.json file (host is omitted):

All configurations (dictionary, engine_settings, grammar) are set to default / are unchanged. My API spec uses token based authentication and I have a simple script to generate the token:

print("{u'myapp': {}}")
print("Authorization: Bearer XXXXXX")

Even when trying to force other errors I still receive a 301. For example I tried adding an endpoint with a URI that clearly doesn't exist hoping to get a 401 error but I still receive a 301. Even when I remove the auth token script I expect to receive some failed auth status but I do not.

When I try these endpoints in Postman they are successful so I am not sure where the issue is. I noticed that RESTler does not support API endpoints that depend on server-side redirecting but the endpoint I am using does not.

Any tips on how to troubleshoot this would be appreciated and if there is any information you would like me to provide, let me know.

Summary of the Pull Request

Currently, scripts to generate authentication tokens can only provide one single authentication header for API requests. For various use cases (such as API sandboxes), multiple authentication headers are required to allow the API to function properly (e.g. additional API keys).

This PR allows authentication scripts to provide multiple authentication headers for all requests, where the required headers for each application are separated with a --- delimiter on a new line.

The new format for authentication scripts requiring multiple authentication headers for each request is:

{u'app1': {}, u'app2':{}}
ApiTokenTag: 9A
ApiTokenTag2: 85
---
ApiTokenTag: ZQ
ApiTokenTag2: UE

This change is backwards compatible, since headers are still parsed for each application if the --- delimiter is not found:

{u'app1': {}, u'app2':{}}
ApiTokenTag: 9A  // Header for app1
ApiTokenTag: ZQ  // Header for app2

The changes have been manually tested on our real-world API example and it worked as expected. Also a separate unit test is included to cover all edge cases: https://github.com/microsoft/restler-fuzzer/pull/358/files#diff-bf3784fc76cc224802461f3a4b53e6a920368c5b39d805edbd30bfe9ee2ed35bR6

PR Checklist

• [x] CLA signed. If not, go over here and sign.
• [x] Tests added/passed.
• [x] Requires documentation to be updated
• [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different approach. Issue number where discussion took place: #xxx

Hi RESTler authors,

I'm using RESTler to test a small local-hosted service, for which RESTler doesn't seem to infer a lot of producer-consumer dependencies (please see attached dependencies.json.txt). In the fuzzing phase, however, I still see a lot of sequences of requests generated in various lengths (please see attached network.testing*.txt). I wonder why? If there is no dependency relationship, would a request be appended to any existing sequence? Is it because of checkers? Thanks! dependencies.json.txt network.testing.140060271036160.1.txt

Starting restler with cmd:

restler test --grammar_file Compile/grammar.py --dictionary_file Compile/dict.json --token_refresh_interval 3500 --token_refresh_command "python3.8 /opt/api/some-service/token-some-service.py" --no_ssl

Some minutes later:

ERROR: Restler engine failed. See logs in /opt/api/admin-service/Test directory for more information.

But EngineStdErr is empty. The information in EngineStdOut:

Initializing: Garbage collection every 30 seconds. Terminating garbage collection. Waiting for max 300 seconds. 'utf-8' codec can't decode byte 0x9d in position 681: invalid start byte

The information from main.txt:

Rendering request 14 from scratch

2021-03-18 11:09:59.824: Final Swagger spec coverage: 4 / 25 2021-03-18 11:09:59.824: Rendered requests: 0 / 25 2021-03-18 11:09:59.824: Rendered requests with "valid" status codes: 4 / 0 2021-03-18 11:09:59.824: Num fully valid requests (no resource creation failures): 4 2021-03-18 11:09:59.824: Num requests not rendered due to invalid sequence re-renders: 0 2021-03-18 11:09:59.824: Num invalid requests caused by failed resource creations: 0 2021-03-18 11:09:59.824: Total Creations of Dyn Objects: 0 2021-03-18 11:09:59.824: Total Requests Sent: {'gc': 0, 'main_driver': 71} 2021-03-18 11:09:59.824: Bug Buckets: {'main_driver_504': 1}

Description

RESTler is stripping query from endpoints when trying different combinations for an endpoint (the last combination). In HTTP traffic logs (network.testing.xxxx.txt), the log for such requests has the query present as 'restler_static_string' but it is not part of the actual request sent. Here is the log for an example request.

This makes the errors in speccov.json very misleading because speccov.json includes the last request combination for an endpoint and the last combination is the one with its query stripped.

The following is the log in speccov.json for the same request. The error message says that the mandatory HTTP header "x-ms-blob-type" is not specified for this request. This error does not correspond to the endpoint that is being tested i.e., "/{containerName}/{blob}?comp=seal" because it is generated for the endpoint "/{containerName}/{blob}", without query.

["29431971b3bea3460dbcf6ad5662cac3e7783dad_2": {
        "verb": "PUT",
        "endpoint": `"/{containerName}/{blob}?comp=seal",
        "verb_endpoint": "PUT /{containerName}/{blob}?comp=seal",
        "valid": 0,
        "matching_prefix": "None",
        "invalid_due_to_sequence_failure": 0,
        "invalid_due_to_resource_failure": 0,
        "invalid_due_to_parser_failure": 0,
        "invalid_due_to_500": 0,
        "status_code": "400",
        "status_text": `"An HTTP header that's mandatory for this request is not specified.",
        "error_message": "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that&apos;s mandatory for this request is not specified.\nRequestId:a3243340-f01e-009c-323f-145ad8000000\nTime:2022-12-20T06:54:12.4169971Z</Message><HeaderName>x-ms-blob-type</HeaderName></Error>",
        "request_order": 6,
        "sample_request": {
            "request_sent_timestamp": null,
            "response_received_timestamp": "2022-12-20 06:54:12",
            "request_verb": "PUT",
            "request_uri": "/mycontainer905e6e8e45/test_file",
            "request_headers": [
                "Accept: application/json",
                "Host: x.blob.core.windows.net",
                "x-ms-version: 2021-12-02",
                "AUTHORIZATION TOKEN"
            ],
            "request_body": null,
            "response_status_code": "400",
            "response_status_text": "An HTTP header that's mandatory for this request is not specified.",
            "response_headers": [
                "Content-Length: 301",
                "Content-Type: application/xml",
                "Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0",
                "x-ms-error-code: MissingRequiredHeader",
                "x-ms-request-id: a3243340-f01e-009c-323f-145ad8000000",
                "x-ms-version: 2021-12-02",
                "Date: Tue, 20 Dec 2022 06:54:12 GMT"
            ],
            "response_body": "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that&apos;s mandatory for this request is not specified.\nRequestId:a3243340-f01e-009c-323f-145ad8000000\nTime:2022-12-20T06:54:12.4169971Z</Message><HeaderName>x-ms-blob-type</HeaderName></Error>"
        },
        "tracked_parameters": {}
   }]

Steps to reproduce

• Use RESTler to compile Azure Storage swagger specs.
• Run RESTler in Test mode.

Expected results

Query should not be removed from endpoints.

Actual results

Query removed from combinations of certain requests.

Environment details

OS: Windows, Python: 3.8.2, .NET version: 6.0.404, RESTler version: 9.1.0

Description

Is there a way to exclude certain headers/query params/body from being fuzzed and keep it static as part of the requests, perhaps by changing the config.json or dict.json?

Thanks

💡 Idea

Today, if a resource is specified in 'create_once', it is excluded from fuzzing. This is inconvenient, since the user must now configure two separate RESTler jobs - one with 'create_once' specified, and one just with the creation API so it can be fuzzed.

RESTler's interface should be improved to allow specifying both to fuzz the create API and subsequently use it in 'create_once' mode for dependencies.

Design Notes

No response

Today, it can be slow and painful to figure out some basic questions during coverage investigations by reading the RESTler output files, without writing additional scripts. For example:

1. Which request, if fixed, would unblock the highest number of requests? In other words, which request should I work on fixing first?

2. What are all the failed requests that were actually tried (not skipped because of a dependency)?

3. What is the sequence of concrete requests that were sent for each failure? Today, this needs to be gathered by navigating and copy-pasting in speccov.json, or by finding the same sequence in the network logs (which is not easy to correlate with the speccov file, since request/sequence IDs are not printed there).

This change includes the following improvements to address these issues:

1. The engine writes a second file in addition to speccov.json, speccov-min.json, which includes the raw request and response. This leaves the speccov.json file unmodified, since speccov.json format is optimized for directly uploading to a database for further analysis, whereas speccov-min is generated to provide a quick way to generate a report that is more user-friendly to inspect directly.

2. The engine now prints all combinations to the speccov.json file (up to a maximum), then post-processes this file to determine the data for each request, and provides the first request as the sample request. This allows failures for user-specified examples to be quickly investigated - the last request, on the contrary, is usually a RESTler-generated request (e.g. with 'fuzzstring' plugged in for all string parameters).

Closes #637

3. This new 'speccov-min.json' is post-processed by the RESTler driver to produce a new text file, which contains the failed requests, prioritized by the number of dependent requests. For each failed request, RESTler includes the full sequence of requests and responses up to the failure.

4. Fixed several bugs and missing information in the spec coverage file.

Description

The endpoint should be taken from the request ID in the grammar.

The current implementation does not work for x-ms-paths - the endpoint should include the query in those cases.

Steps to reproduce

Provide an endpoint in per-resource settings. Then, inspect the request collection endpoints - they do not correspond to the endpoints specified in the grammar.

Expected results

No response

Actual results

No response

Environment details

No response