This is still incomplete, but core and flaskparser tests are now passing. No work is (yet) included on other concrete parsers, but I fixed the base asyncparser to make mypy linting pass.

Resolves #419, #267, #164, #268 Obsoletes, and therefore closes, #410

I've updated the changelog with a basic "statement of intent". As I work on this, I will try to maintain it as an accurate record of how this change should look to the outside world. Obviously, narrative docs also need to be updated with relevant info.

Highlights / greatest hits:

• Marks this as an unreleased v6.0.0 in the changelog
• Replace DEFAULT_LOCATIONS with DEFAULT_LOCATION="json"
• Replace parse_{location} with location_load_{location}
• Replace locations=<iterable> with location=<str>
• Remove locations=... from Fields
• Remove Parser.parse_arg
• Remove webargs.core.get_value
• Add webargs.multidictproxy.MultiDictProxy which has get_value-like behavior and proxies various dict-like objects
• Reverse the decision made in #297 with a comment in tests explaining this change and a bit of similar explanation in the commit message
• Numerous fixes to tests to be more explicit about which location is being used (since you can't trivially have a single location /echo which loads data from a bunch of locations)
• As a side-effect of the above changes, certain inputs which were previously ignored are treated as errors (e.g. 1 for a JSON body will now parse and be passed literally to the schema, which will fail because 1 is not a valid input for schema.load). Changes to the tests serve as a somewhat-readable list of behavioral changes.

To try to keep this from sprawling even more, my intent is to address some items mentioned in #419 either in follow-up PRs or as part of this only if we're happy with this direction (and agree on those items). In particular: the json_or_form location I proposed and nested use_args usage and error collection.

@sloria, @lafrech, my biggest question is: do you like where this is going? Should I just hammer out all of the other parsers?

Now that marshmallow 3.x is in beta and comes with configurable behavior on extra args, how about just including all fields in Parser._parse_request() and passing them on to Schema.load()? It would be up to passed Schema to determine what to do with them.

The effect would be that:

• With RAISE, extra fields would be denied.
• With INCLUDE, extra fields would be allowed and passed on.
• With EXCLUDE, extra fields would be ignored.

This would allow something like:

@use_args(HeaderSchema(unknown=EXCLUDE), locations=('headers', ))
@use_args({'action': fields.Str()}, locations=('json',))  # default is RAISE

Actually, I don't think this even needs marshmallow 3.x. Why not just always pass all fields from _parse_request() to Schema.load()? On 2.x you would need custom @validates_schema implementation in addition, but on 3.x you would not.

Maybe I'm missing something...

Greetings, @sloria

My apologies in case this is not the right channel to address this. I'm a designer in development and an open source enthusiast, who exploring GitHub found your project and decided to propose a logo design for it. It's (of course) totally free and we would be working together to create the design that fits best. In case you agree, you could share some ideas you may have (colours, shapes, etc) so I have something to start with.

Kind regards and keep up the great work!

Hello,

Your recent release of 6.0.0 of webargs has unexpectedly broken our Flask apps. It seems now the @use_kwargs decorator from the webargs.flaskparser completely fails to parse GET query string parameters or POST form-data parameters. Only JSON request bodies ever get parsed anymore.

Here is an example Flask app that shows the problem:

from flask import Flask, jsonify
from webargs import fields
from webargs.flaskparser import use_kwargs

app = Flask(__name__)

@app.route("/test1", methods=["GET", "POST", "PUT"])
@use_kwargs({
    "page": fields.Int(missing=1),
    "per_page": fields.Int(missing=20),
    "full": fields.Bool(missing=False),
})
def test1(**kwargs):
    return jsonify(kwargs)

@app.route("/test2", methods=["GET", "POST", "PUT"])
@use_kwargs({
    "name": fields.Str(required=True),
})
def test2(**kwargs):
    return jsonify(kwargs)

app.run()

The endpoint "/test1" has three optional parameters each with default values when missing. If I make a GET request (with query parameters) or a POST request with application/x-www-form-urlencoded format, webargs completely fails to pick up my parameters and only sees the defaults.

In the endpoint "/test2" I have a required parameter, which again fails to parse on GET or form-data post but works on JSON post only. Here are some curl examples:

# GET request doesn't parse any param
% curl 'http://localhost:5000/test1?page=5&per_page=2&full=1'
{"full":false,"page":1,"per_page":20}

# Normal POST doesn't parse any param
% curl -X POST -d page=5 -d per_page=2 -d full=1 'http://localhost:5000/test1'
{"full":false,"page":1,"per_page":20}

# JSON POST actually does parse params
% curl -X POST -H 'Content-Type: application/json' -d '{"page": 5, "per_page": 2, "full": true}' 'http://localhost:5000/test1'
{"full":true,"page":5,"per_page":2}

# GET is "missing required parameter"
% curl 'http://localhost:5000/test2?name=alice'
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>422 Unprocessable Entity</title>
<h1>Unprocessable Entity</h1>
<p>The request was well-formed but was unable to be followed due to semantic errors.</p>

# POST is "missing required parameter"
% curl -X POST -d name=alice 'http://localhost:5000/test2'
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>422 Unprocessable Entity</title>
<h1>Unprocessable Entity</h1>
<p>The request was well-formed but was unable to be followed due to semantic errors.</p>

# POST JSON actually works
% curl -X POST -H 'Content-Type: application/json' -d '{"name": "alice"}' 'http://localhost:5000/test2'
{"name":"alice"}

I just noticed that something in webargs or marshmallow isn't thread-safe. Take this minimal example"

import time

from flask import Flask, jsonify, request
from marshmallow.fields import Field
from webargs.flaskparser import use_kwargs


app = Flask(__name__)


class MyField(Field):
    def _serialize(self, value, attr, obj):
        return value

    def _deserialize(self, value, attr, data):
        print 'deserialize', request.json, value
        time.sleep(0.25)
        return value


@app.route('/test', methods=('POST',))
@use_kwargs({
    'value': MyField(),
})
def test(value):
    time.sleep(1)
    return jsonify(webargs_result=value, original_data=request.json['value'])

Run it with threading enabled:

$ FLASK_APP=webargsrace.py flask run -p 8080 --with-threads

Now send two requests in parallel, with different values:

$ http post http://127.0.0.1:8080/test 'value=foo' & ; http post http://127.0.0.1:8080/test 'value=bar' &

The output from these two requests is:

{
    "original_data": "bar",
    "webargs_result": "bar"
}

{
    "original_data": "foo",
    "webargs_result": "bar"
}

Clearly not what one would have expected! :bomb:

The output of the print statement showing the request data and what the field receives confirms the issue:

deserialize {u'value': u'bar'} bar
deserialize {u'value': u'foo'} bar

Tested with the latest marshmallow/webargs from PyPI, and also the marshmallow3 rc (marshmallow==3.0.0rc4, webargs==5.1.2).

• Setup GH Actions build for testing, mypy, and twine check
• Add check-jsonschema to pre-commit config
• Remove Azure Pipelines

I originally tried this with the GitHub reusable workflows feature, but a reusable workflow can't be run under a matrix config.

Two more things I'd like us to do on the CI checklist:

• Turn on ReadTheDocs PR builds (Settings > Advanced > PR builds).
• We need a pypi token from someone with push access to setup a tag->publish workflow, set as a repo secret. Without this, we can't create a GH Actions publishing job.

I hesitated to introduce check-jsonschema, as it is my own work, but I find it useful. It catches a wide class of invalid workflow files when tweaking builds. If there's any concern about pulling it in, I'll drop it from the config, as it's not a big deal.

I've initiated a Marshmallow 3 port. Feedback welcome.

I'm not sure how to let tox/Travis test on both Marshmallow versions.

Regarding test_error_handler_is_called_regardless_of_schema_strict_setting, see https://github.com/sloria/webargs/issues/187.

When a content of a request is read before parser.parse is called (with location=('json',)), parser returns that JSON fields are missing in the request.

Example code:

async def handler(request):
    data = await request.json()
    args = await parser.parse(handler_args, req=request, locations=('json',))
    return web.Response()

The issue is the following line of code https://github.com/sloria/webargs/blob/dev/webargs/aiohttpparser.py#L92 specifically req.has_body. This attribute has an unfortunate name and it's not constant over a lifetime of a request. It checks if there are more bytes to read. Once the body is read by await request.json() there are no more bytes to read and value of req.has_body is False.

The docs for the fields.DelimitedList say it "can load from either a list or a delimited string" but I cannot get it to load from a list. It's possible I'm doing something wrong but When I swap out fields.DelimitedList for fields.List everything works as expected. I think either the docs should be changed, or we should fix the implementation of DelimitedList so that it works as expected.

I would like my API to offer basic query features, like not only get item by ID but also using operators on item attributes:

GET /resource/?name=chuck&surname__contains=orris&age__lt=69

Proof of concept

Here is what I've done so far. I did not modify webargs code. I'm only subclassing in my own application.

I've been searching around and didn't find a universally accepted norm specifying such a query language. In my implementation, I'm using the double underscore syntax and a subset of operators from MongoEngine.

Basically, I have two families of operators, some operating on numbers, others on string.

NUMBER_OPERATORS = ('ne', 'gt', 'gte', 'lt', 'lte')

STRING_OPERATORS = (
    'contains', 'icontains', 'startswith', 'istartswith',
    'endswith', 'iendswith', 'iexact'
)

QUERY_OPERATORS = {
    'number': NUMBER_OPERATORS,
    'string': STRING_OPERATORS,
}

Those lists could probably be extended. Operators meanings should be easy to grap. Examples:

• age__lt=69 means I expect the API to return records with attribute age lower than 69
• surname__contains=orris means attribute age contains string "orris"

To let webargs parse such parameters, I need to modify the Schema so that for chosen fields, the Marshmallow field is duplicated (deepcopied) into needed variants. For instance, the field age is duplicated into age__lt, age__gt,...

This is an opt-in feature. For each field I want to expose this way, I need to specify in Meta which operators category it should use (currently only two categories: number and string). Auto-detection is complicated as I don't know how I would handle custom fields, and there may be other categories some day.

In the Schema, I add:

    class Meta:
        fields_filters = {
            'name': ('string',),
            'surname': ('string',),
            'age': ('number',),
        }

For each field, I'm passing a list of categories as one could imagine several categories applying to a field, but currently, I have no example of field that would use both number and string operators.

And the "magic" takes place here:

class SchemaOpts(ma.SchemaOpts):
    def __init__(self, meta):
        super(SchemaOpts, self).__init__(meta)
        # Add a new meta field to pass the list of filters
        self.fields_filters = getattr(meta, 'fields_filters', None)


class SchemaMeta(ma.schema.SchemaMeta):
    """Metaclass for `ModelSchema`."""

    @classmethod
    def get_declared_fields(mcs, klass, *args, **kwargs):

        # Create empty dict using provided dict_class
        declared_fields = kwargs.get('dict_class', dict)()

        # Add base fields
        base_fields = super(SchemaMeta, mcs).get_declared_fields(
            klass, *args, **kwargs
        )
        declared_fields.update(base_fields)

        # Get allowed filters from Meta and create filters
        opts = klass.opts
        fields_filters = getattr(opts, 'fields_filters', None)

        if fields_filters:
            filter_fields = {}
            for field_name, field_filters in fields_filters.items():
                field = base_fields.get(field_name, None)
                if field:
                    for filter_category in field_filters:
                        for operator in QUERY_OPERATORS.get(
                                filter_category, ()):
                            filter_fields[
                                '{}__{}'.format(field_name, operator)
                            ] = deepcopy(field)
            declared_fields.update(filter_fields)

        return declared_fields


class QueryArgsSchema(ma.compat.with_metaclass(SchemaMeta, ma.Schema)):
    OPTIONS_CLASS = SchemaOpts

And finally, I use the Schema to parse the query arguments:

    @use_args(ObjectSchema)
    def get(self, args):
        ...

Questions

This raises a few points.

• Is there some sort of convention I missed when searching for a query language?
• Is this out-of-scope for webargs or could it be a useful enhancement?
• Is there no need for that? I've been investigating both flask-retful and marshmallow/webargs/... ecosystems, along with @frol's invaluable flask-restplus-server-example and saw nothing close to this, so I'm thinking maybe people just don't do that. Or maybe they only expose a few filters, and they do it in specific routes.
• Should this be in @touilleMan's marshmallow-mongoengine? I do use this library, but although the query language is inspired^Wshamelessly copied from MongoEngine (which allows me to pass the query arguments straight into the QuerySet filters...), the whole thing has no dependency on MongoEngine and this should be a generic feature.
• My QueryArgsSchema also has sort and pagination fields, but I didn't expose them here as this needs nothing fancy on Marshmallow's side. Maybe a real "query parameters" feature would integrate these as well.

Feedback greatly appreciated. It seems to work right now, but on the long run, I might discover it was poorly designed from the start.

Thanks.

Hey guys,

I am having an issue with webargs==5.x, in that, when I upgrade it from 4.x to 5.x, using Flask==1.0.2, it works and behaves normally on python 3.7, but it behaves differently on python 2.7 on the same code base. The issue is on this pull request mostafa/grest#84 and is evident on this build: https://travis-ci.org/mostafa/grest/builds/479199700. I've tried to trace my own code to see if there is a path I am not covering, but that seems not to be the case, because it passes all tests on python 3.x.

Thanks, Mostafa.

In the next major webargs release (9.0), we should make sure to update the minimum versions of the various frameworks which we test against.

IIRC, the current bounds came from looking at which versions we support in practice. But we don't want to be stuck testing against old flask, django, pyramid, etc versions indefinitely. We could debate exactly what kinds of updates are safe in a minor release -- especially since this is for our testing configuration and not part of our install_requires -- but I think it's simpler to just do this in 9.0, whenever that may be.

In #687 , we started to discuss our more general setup in terms of Azure Pipelines vs GitHub Actions (GH Actions). A couple of us support the switch to GH Actions, and there's no strong objection to it. I'd like to try that out.

The setup I want to try would be something like a reusable workflow to run tox, in the repo, and then a job named build which runs the tox workflow over a matrix of python version, os, and tox env names. Now that reusable workflows supports using a local file, we can develop something in webargs until we like the result. If we then want to move it to a separate project like marshmallow-code/reusable-github-workflows, we can discuss and/or do that.

There ~are two notable jobs~ is one job in Azure Pipelines which I don't think belongs in the move to GH Actions:

• tox -e docs
• ~pypi publish~ (done in #690)

For the docs, ReadTheDocs (RTD) offers a relatively new feature to build on PRs; I recall we discussed it when it was still in beta. I like this primarily because it's driven by the .readthedocs.yml in the repo, so we don't run into trouble where our tox -e docs run in CI differs from what happens in RTD. It also means that if we have a PR to update .readthedocs.yml config, it will be tested in the PR build. All we have to do is enable this feature in the readthedocs.org admin pane and make sure that the RTD webhook is set to send Pull Request events. A trial PR could demonstrate that this works -- e.g. by updating the python version used for RTD.

~For pypi publishing, I think we can leave this in Azure for the initial move to GitHub Actions. It should be possible to setup a tag->publish reusable workflow, but I want to get more "basic" CI worked out before taking a crack at this.~ (done in #690)

With #690 now merged, we have a short "remaining" TODO list:

• [ ] Turn on ReadTheDocs PR builds (Settings > Advanced > PR builds)
• [x] Remove testpypi releasing once we've tested that it works
• [ ] evaluate reusable workflows so that we can reduce duplication in config across repos

There are a few cases of interesting usages that come up in issues, but which don't seem appropriate to put into our main docs (b/c we don't want to bloat our docs). If we had an Examples page in the docs, we could show off all of our cool ideas for how you can solve problems with webargs, but without disrupting the flow of the main documentation.

Specific cases which would be good candidates for example apps:

• Django App with CSRF-Removing Parser
• Axios Query-String Parser
• Setting a parser to use unknown=RAISE on query params

I did a quick scan of closed issues until I hit the 5.x/6.0 transition point, then called it a day. Maybe there are other interesting examples in older issues? Or maybe we have things in other parts of the docs like the upgrade guide which would be nice to separate out (or even just duplicate).

The documentation on passed values (https://flask-smorest.readthedocs.io/en/latest/arguments.html) for form values, json, headers, query, etc. do not have case sensitivity documented. I would expect locations like json would be case sensitive, but locations like headers to be case insensitive. Looking at https://github.com/marshmallow-code/flask-smorest/blob/e06325d7a201af6d07caa2e8e103fba15eb4b9b7/flask_smorest/arguments.py#L105 I'd guess everything is case sensitive.

Hi,

I'm using axios to send a GET request to my flask-smorest endpoint. I know there's no real agreement on this spec-wise, but the request's querystring contains an array, and axios sends it in this format:

http://127.0.0.1:5000/api/v1/items/?types[]=cd&types[]=dvd

I've defined the schema used for reading the arguments as

class FilterSchema(ma.Schema):
    ...
    types = ma.fields.List(ma.fields.String(), missing=[])

Yet when I try to read the data received in my endpoint, types is empty:

@items_blp.route('/')
class ItemCollection(MethodView):
    @items_blp.arguments(FilterSchema(unknown=ma.EXCLUDE), location="query")
    @listings_blp.response(ItemSchema(many=True))
    def get(self, payload):
        # payload['types'] is empty...
        if payload['types']:
            qs = Item.objects.filter(item_type__in=payload['types'])
        return qs

Is this a missing feature in flask-smorest or should I ask on SO whether I should use another way to pass data?

Add a decorator that will parse request arguments from the view function's type annotations.

I've built a working proof of concept of this idea in webargs-starlette.

@app.route("/")
@use_annotations(locations=("query",))
async def index(request, name: str = "World"):
    return JSONResponse({"Hello": name})

A marshmallow Schema is generated from the annotations using Schema.TYPE_MAPPING to construct fields.

The code for use_annotations mostly isn't tied to Starlette and could be adapted for AsyncParser (and core.Parser when we drop Python 2 support).