A lightweight library for converting complex objects to and from simple Python datatypes.

Overview

marshmallow: simplified object serialization

Latest version Build status Documentation code style: black

marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.

from datetime import date
from pprint import pprint

from marshmallow import Schema, fields


class ArtistSchema(Schema):
    name = fields.Str()


class AlbumSchema(Schema):
    title = fields.Str()
    release_date = fields.Date()
    artist = fields.Nested(ArtistSchema())


bowie = dict(name="David Bowie")
album = dict(artist=bowie, title="Hunky Dory", release_date=date(1971, 12, 17))

schema = AlbumSchema()
result = schema.dump(album)
pprint(result, indent=2)
# { 'artist': {'name': 'David Bowie'},
#   'release_date': '1971-12-17',
#   'title': 'Hunky Dory'}

In short, marshmallow schemas can be used to:

  • Validate input data.
  • Deserialize input data to app-level objects.
  • Serialize app-level objects to primitive Python types. The serialized objects can then be rendered to standard formats such as JSON for use in an HTTP API.

Get It Now

$ pip install -U marshmallow

Documentation

Full documentation is available at https://marshmallow.readthedocs.io/ .

Requirements

  • Python >= 3.5

Ecosystem

A list of marshmallow-related libraries can be found at the GitHub wiki here:

https://github.com/marshmallow-code/marshmallow/wiki/Ecosystem

Credits

Contributors

This project exists thanks to all the people who contribute.

You're highly encouraged to participate in marshmallow's development. Check out the Contributing Guidelines to see how you can help.

Thank you to all who have already contributed to marshmallow!

Contributors

Backers

If you find marshmallow useful, please consider supporting the team with a donation. Your donation helps move marshmallow forward.

Thank you to all our backers! [Become a backer]

Backers

Sponsors

Support this project by becoming a sponsor (or ask your company to support this project by becoming a sponsor). Your logo will show up here with a link to your website. [Become a sponsor]

Sponsors Become a sponsor

Professional Support

Professionally-supported marshmallow is now available through the Tidelift Subscription.

Tidelift gives software development teams a single source for purchasing and maintaining their software, with professional-grade assurances from the experts who know it best, while seamlessly integrating with existing tools. [Get professional support]

Get supported marshmallow with Tidelift

Security Contact Information

To report a security vulnerability, please use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.

Project Links

License

MIT licensed. See the bundled LICENSE file for more details.

Comments
  • Add option to throw validation error on extra keys

    Add option to throw validation error on extra keys

    I'm not sure if I'm missing something, but I would like to throw error if extra keys are provided, as in:

    >>> class AlbumSchema(Schema):
    ...     title = fields.Str()
    
    >>> AlbumSchema(strict=True).load({'extra': 2}, allow_extra=False)
    Traceback (most recent call last):
    ...
    marshmallow.exceptions.ValidationError: {'_schema': ["Extra arguments passed: ['extra']"]}
    

    I've been using the following implementation (taken from https://github.com/sloria/webargs/issues/87#issuecomment-183949558):

    class BaseSchema(Schema):
    
        @pre_load
        def validate_extra(self, in_data):
            if not isinstance(in_data, dict):
                return
    
            extra_args = [key for key in in_data.keys() if key not in self.fields]
            if extra_args:
                raise ValidationError('Extra arguments passed: {}'.format(extra_args))
    

    I would expect this to be a common need, however, so could it be supported by the library out-of-the-box?

    enhancement feedback welcome 
    opened by tuukkamustonen 47
  • Field validators as schema methods?

    Field validators as schema methods?

    Often, validators and preprocessors only apply to a single Schema, so it may make sense to define them within the Schema--rather than as separate functions--for better cohesion.

    Method names could be passed to __validators__, et. al, like so:

    class UserSchema(Schema):
        __validators__ = ['validate_schema']
    
        def validate_schema(self, data):
            # ...
    
    enhancement feedback welcome 
    opened by sloria 37
  • Handle unknown fields with EXCLUDE, INCLUDE or RAISE

    Handle unknown fields with EXCLUDE, INCLUDE or RAISE

    This is a rework of #595 so that it supports the API discussed in #524.

    By default, only the fields described in the schema are returned when data is deserialized.

    This commit implements an unknown option, so that Schema(unknown=ALLOW) or Schema().load(data, unknown=ALLOW) permit users to receive the unknown fields from the data.

    Schema(unknown=RAISE) or Schema().load(data, unknown=RAISE) will raise a ValidationError for each unknown field.

    Schema(unknown=IGNORE) or Schema().load(data, unknown=IGNORE) keep the original behavior.

    Edit: ALLOW will finally be INCLUDE, and IGNORE will be EXCLUDE.

    opened by ramnes 36
  • Allow callables for fields.Nested

    Allow callables for fields.Nested

    Suggested earlier here: https://github.com/marshmallow-code/marshmallow/issues/19#issuecomment-43685498.

    The suggestion would be to allow:

    foo = fields.Nested(lambda: Foo)
    

    in addition to

    foo = fields.Nested('Foo')
    

    Seems a bit cleaner in general.

    opened by taion 36
  • Nested 'only' parameter

    Nested 'only' parameter

    This allows to specify a nested only parameter both at schema instantiation time and during dump() like so:

    class ChildSchema(Schema):
        foo = fields.Field()
        bar = fields.Field()
        baz = fields.Field()
    
    class ParentSchema(Schema):
        bla = fields.Field()
        bli = fields.Field()
        blubb = fields.Nested(ChildSchema)
    
    data = dict(bla=1, bli=2, blubb=dict(foo=42, bar=24, baz=242))
    
    # either when instantiating
    sch = ParentSchema(only=('bla', ('blubb', ('foo', 'bar'))))
    result = sch.dump(data)
    
    #or when dumping
    sch = ParentSchema()
    result = sch.dump(data, only=('bla', ('blubb', ('foo', 'bar'))))
    

    It is fully backwards compatible. Fixes #402

    feedback welcome 
    opened by Tim-Erwin 33
  • Processing API for dumping does not match API for loading

    Processing API for dumping does not match API for loading

    When dumping, I want to manipulate some of the attributes on the object being dumped before actually dumping. So I decorate a preprocessor function. But that only gets used when loading. I could use a Method field, but the data returned from that must be the final serialized form, so there's no way to specify that it's actually a Nested(OtherSchema, many=True) unless I do that serialization manually at the end of the method.

    When loading, I want to manipulate the loaded data in the exact opposite direction as the situation above. So I decorate a data_handler function. But that only gets used when dumping. The solution is more straightforward here: I override the make_object method to manipulate the final output.

    My point is that the names are different everywhere, some of them are decorators while others are methods, and they don't apply in both directions, when it would be very convenient to be able to do so. The loading situation is in better shape than the dumping one, since loading has preprocessor and make_object, except there's still a weird decorator vs. method difference.

    There should be a standard way to preprocess and postprocess the entire data during both loading and dumping. One solution is adding pre_dump, post_dump, pre_load, and post_load hooks, either as methods on the schema or as decorators.

    feedback welcome 
    opened by davidism 33
  • Propose add required to schema constructor

    Propose add required to schema constructor

    For a put request model require of id field, but for post request id not needed. For disabling in a post request I use dump_only=["id"], but I have not found such a way to do id required. For this reason, I suggest adding required to the schema constructor.

    question 
    opened by heckad 30
  • How to create a Schema containing a dict of nested Schema?

    How to create a Schema containing a dict of nested Schema?

    Hi. I've been digging around and couldn't find the answer to this.

    Say I've got a model like this:

    class AlbumSchema(Schema):
        year = fields.Int()
    
    class ArtistSchema(Schema):
        name = fields.Str()
        albums = ...
    

    I want albums to be a dict of AlbumSchema, so that ArtistSchema serializes as

    { 'albums': { 'Hunky Dory': {'year': 1971},
                  'The Man Who Sold the World': {'year': 1970}},
      'name': 'David Bowie'}
    

    Naively, I would expect syntaxes like this to work:

    fields.List(Schema)
    fields.Dict(Schema)
    

    or maybe

    fields.List(fields.Nested(Schema))
    fields.Dict(fields.Nested(Schema))
    

    Serializing a list of Schema can be achieved through Nested(Schema, many=True), which I find less intuitive, and I don't know about a dict of Schema.

    Is there any way to do it? Or a good reason not to do it?

    (Question also asked on SO.)

    help wanted needs review 
    opened by lafrech 30
  • Support Enum for Select-field

    Support Enum for Select-field

    I have got some use cases where I use a Enum in my models.

    Example:

    # definitions.py
    from enum import Enum
    
    class Gender(Enum):
        male = 'm'
        female = 'f'
    
    # models.py
    from definitions import Gender
    
    class Person:
        def __init__(self, gender: Gender):
            self.gender = gender
    

    Now it would be great if I could use the Enum in marshmallow.fields.Select:

    # schemas.py
    from marshmallow import fields, Schema
    from definitions import Gender
    from models import Person
    
    class PersonSchema(Schema):
        gender = fields.Select(Gender)
    
        @staticmethod
        def make_object(data) -> Person:
            return Person(**data)
    

    For backwards-compatibility enum34 could be used.

    opened by floqqi 27
  • Validates schema field errors

    Validates schema field errors

    Improves validation error reporting functionality enabling reporting field errors from whole-schema (validates_schema) validator.

    Introduces new public function marshmallow.utils.merge_errors which allows deep merging of errors. It can be useful to simplify accumulating errors in users' validators.

    Updates validation error documentation to clarify error messages format.

    Fixes #441

    backwards incompat 
    opened by maximkulkin 25
  • Implement hooks for a JIT to integrate into marshmallow.

    Implement hooks for a JIT to integrate into marshmallow.

    This is a revival of https://github.com/marshmallow-code/marshmallow/pull/573

    Unfortunately other personal/professional priorities prempted me from focusing on this for the past few months, but I'd like to revisit open sourcing this again.

    When we deployed this to production we saw ~20x improvement in serialization time which, in turn, dropped our response times in half.

    I totally understand the apprehension on the original PR due to the complexity this introduces, so I've created a follow on PR that integrates the hooks I need into marshmallow, leaving everything else relatively unchanged.

    We're still able to get the same performance gains, and I'm able to iterate on my JIT library without having to pester you with PRs :D. If we can get this integration working we'll have an awesome story about how Marshmallow allows engineers to have the amazing API of Marshmallow without having to sacrifice performance.

    opened by rowillia 21
  • Bump sphinx from 5.3.0 to 6.0.0

    Bump sphinx from 5.3.0 to 6.0.0

    Bumps sphinx from 5.3.0 to 6.0.0.

    Release notes

    Sourced from sphinx's releases.

    v6.0.0

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.0b2

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.0b1

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    Changelog

    Sourced from sphinx's changelog.

    Release 6.0.0 (released Dec 29, 2022)

    Dependencies

    • #10468: Drop Python 3.6 support
    • #10470: Drop Python 3.7, Docutils 0.14, Docutils 0.15, Docutils 0.16, and Docutils 0.17 support. Patch by Adam Turner

    Incompatible changes

    • #7405: Removed the jQuery and underscore.js JavaScript frameworks.

      These frameworks are no longer be automatically injected into themes from Sphinx 6.0. If you develop a theme or extension that uses the jQuery, $, or $u global objects, you need to update your JavaScript to modern standards, or use the mitigation below.

      The first option is to use the sphinxcontrib.jquery_ extension, which has been developed by the Sphinx team and contributors. To use this, add sphinxcontrib.jquery to the extensions list in conf.py, or call app.setup_extension("sphinxcontrib.jquery") if you develop a Sphinx theme or extension.

      The second option is to manually ensure that the frameworks are present. To re-add jQuery and underscore.js, you will need to copy jquery.js and underscore.js from the Sphinx repository_ to your static directory, and add the following to your layout.html:

      .. code-block:: html+jinja

      {%- block scripts %} {{ super() }} {%- endblock %}

      .. _sphinxcontrib.jquery: https://github.com/sphinx-contrib/jquery/

      Patch by Adam Turner.

    • #10471, #10565: Removed deprecated APIs scheduled for removal in Sphinx 6.0. See :ref:dev-deprecated-apis for details. Patch by Adam Turner.

    • #10901: C Domain: Remove support for parsing pre-v3 style type directives and roles. Also remove associated configuration variables c_allow_pre_v3 and c_warn_on_allowed_pre_v3. Patch by Adam Turner.

    Features added

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependency 
    opened by dependabot[bot] 0
  • Use fromisoformat from standard library (or backport)

    Use fromisoformat from standard library (or backport)

    This PR shows what it would look like if we used Python 3.11's improved fromisoformat (backported for older releases).

    Less code on our side and it should be more efficient (C implementation).

    (We could simplify even more by passing those functions directly to the class and remove everything from utils.py.)

    There are 2 failing tests.

    • "2018-01-01" is accepted as a datetime while it is currently rejected in marshmallow. I guess this would be a breaking change. We may keep the old behaviour with a temporary trick until marshmallow 4 (keep the regex or even just check input string length).

    • Time deserialization respects timezones. It is not clear to me what a time with timezone but without date means. It could be ambiguous for timezones with DST. In any case, this can be considered either bugfix or breaking change, and it could be "fixed" to keep old behaviour.


    Consider this as an RFC.

    Meanwhile, anyone willing to do it in their app may set those functions as class attributes and use the backport if needed.

    opened by lafrech 1
  • Add new argument to field

    Add new argument to field "do_not_dump_if_null"

    I have a quite big application that grows everyday and I begin to face performance problems both with network loading and sometimes in the frontend. I would therefore like to minimize some of the objects such that I don't pass the attributes if they are null.

    from marshmallow import Schema, fields
    
    class PersonSchema(Schema):
        id = fields.Number()
        name = fields.Str()
        child_dream_job = fields.Str(do_not_dump_if_null=True)
    
    
    album = dict( id =1, name="test", child_dream_job=None)
    
    schema = PersonSchema()
    assert schema.dump(album) == {"id":1, "name":"test"}
    

    So in typescript notation:

    interface IPerson {
      id: number;
      name: string;
      child_dream_job? : string;
    }
    

    instead of

    interface IPerson {
      id: number;
      name: string;
      child_dream_job : string | null;
    }
    
    opened by PMLP-novo 3
  • Emit warning when user passes string instead of list to OneOf

    Emit warning when user passes string instead of list to OneOf

    I just got caught.

    I wrote

        validate=validate.OneOf("one", "two")
    

    instead of

        validate=validate.OneOf(["one", "two"])
    

    "one" would pass because "one" in "one" is True.

    It might save users trouble if we emitted a warning when a string is passed instead of a non-string iterable.

    In practice, it only affects lists made of two elements (or one but users would use Equal), so no big deal.

    enhancement 
    opened by lafrech 2
  • Include a nested relation to one schema, without modifying the rest of the functions that use it

    Include a nested relation to one schema, without modifying the rest of the functions that use it

    Basically I would like to add a new nested relation to an schema, that I already use in other services, without this implying a change to the rest of the functionalities that use it for serialization. Since I work with sqlalchemy, if I include the new relation, new unwanted statements appears everywhere, since the ORM tries to fill it. I would need something like adding the relation, and specifying that be only included if I specify it. I seen this kind of problem resolved in other libraries using groups annotations, so in this case, the relation is added, annotated with a new group, and them call serialize using the existing groups plus the new one.

    Note: Create a new schema is not a good solution since the affected schema is deep in the relation, so I would have to create all it's fathers too.

    opened by abdielcs 0
Owner
marshmallow-code
Python object serialization and deserialization, lightweight and fluffy
marshmallow-code
Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

jsonpickle jsonpickle is a library for the two-way conversion of complex Python objects and JSON. jsonpickle builds upon the existing JSON encoders, s

null 1.1k Jan 2, 2023
Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy

orjson orjson is a fast, correct JSON library for Python. It benchmarks as the fastest Python library for JSON and is more correct than the standard j

null 4.1k Dec 30, 2022
Generic ASN.1 library for Python

ASN.1 library for Python This is a free and open source implementation of ASN.1 types and codecs as a Python package. It has been first written to sup

Ilya Etingof 223 Dec 11, 2022
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON <http://json.org> encoder and decoder for Python 3.3+ with legacy suppo

null 1.5k Dec 31, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.6k Jan 1, 2023
MessagePack serializer implementation for Python msgpack.org[Python]

MessagePack for Python What's this MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JS

MessagePack 1.7k Dec 29, 2022
Ultra fast JSON decoder and encoder written in C with Python bindings

UltraJSON UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.6+. Install with pip: $ python -m pip insta

null 3.9k Jan 2, 2023
Crappy tool to convert .scw files to .json and and vice versa.

SCW-JSON-TOOL Crappy tool to convert .scw files to .json and vice versa. How to use Run main.py file with two arguments: python main.py <scw2json or j

Fred31 5 May 14, 2021
serialize all of python

dill serialize all of python About Dill dill extends python's pickle module for serializing and de-serializing python objects to the majority of the b

The UQ Foundation 1.8k Jan 7, 2023
🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

srsly: Modern high-performance serialization utilities for Python This package bundles some of the best Python serialization libraries into one standa

Explosion 329 Dec 28, 2022
Python wrapper around rapidjson

python-rapidjson Python wrapper around RapidJSON Authors: Ken Robbins <[email protected]> Lele Gaifax <[email protected]> License: MIT License Sta

null 469 Jan 4, 2023
Python bindings for the simdjson project.

pysimdjson Python bindings for the simdjson project, a SIMD-accelerated JSON parser. If SIMD instructions are unavailable a fallback parser is used, m

Tyler Kennedy 562 Jan 8, 2023
A lightweight library for converting complex objects to and from simple Python datatypes.

marshmallow: simplified object serialization marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, t

marshmallow-code 6.4k Jan 2, 2023
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021
Python-zhuyin - An open source Python library that provides a unified interface for converting between Chinese pinyin and Zhuyin (bopomofo)

Python-zhuyin - An open source Python library that provides a unified interface for converting between Chinese pinyin and Zhuyin (bopomofo)

null 2 Dec 29, 2022
Library for converting from RGB / GrayScale image to base64 and back.

Library for converting RGB / Grayscale numpy images from to base64 and back. Installation pip install -U image_to_base_64 Conversion RGB to base 64 b

Vladimir Iglovikov 16 Aug 28, 2022
3DigitDev 29 Jan 17, 2022
Lightweight mmm - Lightweight (Bayesian) Media Mix Model

Lightweight (Bayesian) Media Mix Model This is not an official Google product. L

Google 342 Jan 3, 2023
A library for converting HTML into PDFs using ReportLab

XHTML2PDF The current release of xhtml2pdf is xhtml2pdf 0.2.5. Release Notes can be found here: Release Notes As with all open-source software, its us

null 2k Dec 27, 2022
peartree: A library for converting transit data into a directed graph for sketch network analysis.

peartree ?? ?? peartree is a library for converting GTFS feed schedules into a representative directed network graph. The tool uses Partridge to conve

Kuan Butts 183 Dec 29, 2022