A simple, fast, extensible python library for data validation.

Last update: Sep 19, 2022

Related tags

Overview

Validr

A simple, fast, extensible python library for data validation.

Simple and readable schema
10X faster than jsonschema, 40X faster than schematics
Can validate and serialize any object
Easy to create custom validators
Accurate and friendly error messages

简单，快速，可拓展的数据校验库。

简洁，易读的 Schema
比 jsonschema 快 10 倍，比 schematics 快 40 倍
能够校验&序列化任意类型对象
易于拓展自定义校验器
准确友好的错误提示

Overview

from validr import T, modelclass, asdict

@modelclass
class Model:
    """Base Model"""

class Person(Model):
    name=T.str.maxlen(16).desc('at most 16 chars')
    website=T.url.optional.desc('website is optional')

guyskk = Person(name='guyskk', website='https://github.com/guyskk')
print(asdict(guyskk))

Install

Note: Only support Python 3.5+

pip install validr

When you have c compiler in your system, validr will be c speedup mode. Otherwise validr will fallback to pure python mode.

To force c speedup mode:

VALIDR_SETUP_MODE=c pip install validr

To force pure python mode:

VALIDR_SETUP_MODE=py pip install validr

Document

https://github.com/guyskk/validr/wiki

Performance

benchmark result in Travis-CI:

--------------------------timeits---------------------------
  voluptuous:default             10000 loops cost 0.368s
      schema:default              1000 loops cost 0.318s
        json:loads-dumps        100000 loops cost 1.380s
      validr:default            100000 loops cost 0.719s
      validr:model              100000 loops cost 1.676s
  jsonschema:draft3              10000 loops cost 0.822s
  jsonschema:draft4              10000 loops cost 0.785s
  schematics:default              1000 loops cost 0.792s
---------------------------scores---------------------------
  voluptuous:default               375
      schema:default                43
        json:loads-dumps          1000
      validr:default              1918
      validr:model                 823
  jsonschema:draft3                168
  jsonschema:draft4                176
  schematics:default                17

Develop

Validr is implemented by Cython since v0.14.0, it's 5X faster than pure Python implemented.

setup:

It's better to use virtualenv or similar tools to create isolated Python environment for develop.

After that, install dependencys:

./bootstrap.sh

build, test and benchmark:

inv build
inv test
inv benchmark

License

Anti-996 License

Comments

list.unique validator is SLOW

I have a schema like

plain_rule_schema_in = T.dict(
    enabled=T.bool.optional.default,
    ttl=T.int.optional,
    settings=T.dict(
        addr=T.netaddr
    )
)

plain_rule_list_in = T.dict(
    name=T.str,
    enabled=T.bool,
    description=T.str.optional.default(''),
    category=T.enum(
        ','.join([category.name for category in RuleListCategoriesEnum])
    ),
    rules=T.list(
        plain_rule_schema_in,
    ).maxlen(100000)
)

of a relatively big list of rule entities inside of the list container. It processes fast enough (0.23 sec for 20000 entities), but that's before i add a unique checker to the list. Time instantly jumps to 15-16 s! I can imagine that unique check is a little slow, yes, but not SUCH slow. There may be a catch here, of course, for example, we can try to drop list/dict objects to json, and many others to str to do substantially faster checks. You may think of something even faster, but for now, this great check is literally unusable I use validr==1.0.4, netaddr check is performed via

@validator(string=True)
def netaddr_validator(compiler):
    """Custom validator

    Params:
        compiler: can be used for compile inner schema
        items: optional, and can only be scalar type, passed by schema in `T.validator(items)` form
        some_param: other params
    Returns:
        validate function
    """
    def validate(value):
        """Validate function

        Params:
            value: data to be validate
        Returns:
            valid value or converted value
        Raises:
            Invalid: value invalid
        """
        try:
            value = str(netaddr.IPNetwork(value))
        except netaddr.core.AddrFormatError as ex:
            raise Invalid('{} is invalid addr'.format(value))

        return value
    return validate

opened by survivorm 14

fix list unique check slow. #17
The unique check is O(1) operation for each item, the key function is based on inner schema. Only list of types below are available:

scalar type, eg: bool, int, float, str

list of scalar type

dict of scalar type value

other types will cause SchemaError.
opened by guyskk 9
Example request: Nesting validators

Is there a way to create custom validator with the ability to use nested other validators (as list and dict do?) I know i'm using the validr in an unexpected way, but the case is: I have the db results which i want to pack into arbitrary data structure. One of the fields of the list of nested dicts (representing db rows) is bytea in postgre (python bytes), packed json dump -> str -> bytes. So, i need to unpack it, and ideally - check the result as a dict. Is it possible?

opened by survivorm 6

Feature/add helper funcs

You may specify general purpose helper functions for the compiler like this:

compiled = Compiler(
                        validators={
                            'enum': enum_validator,
                            'netaddr': netaddr_validator
                        },
                        helpers={
                            'rule_unique': rule_key_func
                        }
                    ).compile(plain_rule_list_in)

and then use them in list checker

from validr import T, modelclass, asdict, validator, Compiler

plain_rule_schema_in = T.dict(
    enabled=T.bool.optional.default,
    ttl=T.int.optional,
    settings=T.dict(
        addr=T.netaddr
    )
)

plain_rule_list_in = T.dict(
    name=T.str,
    enabled=T.bool,
    description=T.str.optional.default(''),
    category=T.enum(
        ','.join([category.name for category in RuleListCategoriesEnum])
    ),
    rules=T.list(
        plain_rule_schema_in,
    ).maxlen(100000).unique.optional.unique_key_helper('rule_unique')
)

helper function value must be unique for each list entity

e.g:

def rule_key_func(value):
    return json.dumps(value)

opened by survivorm 4

Can't call with keyword argument

I am trying out a simple schema and I realize it is not possible to use optional while having additional attributes declared in a dict schema. Am I doing it correctly? I cannot find any relevant example regarding this usage.

As such,

from validr import T, modelclass, asdict, ValidrError

@modelclass
class Model:
    """Base Model"""

class Person(Model):

    url=T.url.desc("url")
    myObj = T.dict.optional(
        idx=T.int.optional.default(0)
    )

try:
    result = Person(url="http://test.com", myObj={
        "idx": 1
    })
    print(asdict(result))
except ValidrError as err:
    print(err.message)

opened by rexlow 3

Allow option to throw Invalid exception as either AsciiTable or dictionary.
Validr is currently unsupported for server applications that require verifying POST requests. Since Validr has implemented AsciiTable as opposed to either a dictionary or a list of errors, when creating RESTful API's the thrown exception is absolutely useless.

A better implementation would perhaps be the ability to define some variable within the base Model classes to give better control of the error:

@modelclass class Model: self.error_type = dict class Person(Model): name=T.str.maxlen(16).desc('at most 16 chars') website=T.url.optional.desc('website is optional')

This way when an error is found, the server could easily return back what is going on.

try: test = Person(name=True, website='') except Exception as e: return json({'error': 'Invalid key(s) input.', 'keys': e})

Where the error e would be:

{'name': 'invalid string'}

And therefore the server is able to respond:

{ 'error': 'Invalid key(s) input.', 'keys': { 'name': 'invalid string' } }

For now, unfortunately, I will have to go back to another json verifier. Great project nonetheless!
opened by synchronizing 3

New schema syntax discussion

Goals

Easy to write schema in python/high-level programming language, fewer mistakes
Easy to share and refer schema, and define openapi
Compatible with YAML, replace & with . and remove @ grammar

Overview:

# old
name?str&strip&default="world"&desc="Your name"
# new-yaml
name: str.strip.default="world".desc="Your name"
# new-python
name: T.str.strip.default("world").desc("your name")
# new refer
pet: T.ref("http://example.com/schema.json#Pet").optional.desc('description')

Syntax in YAML

scalar:
    validator.bool.key=value
list:
    - validator.bool.key=value
    - arg0
    - arg1
dict:
    $self: validator.bool.key=value
    key0: value
    key1: value
refer:
    pet: ref("http://example.com/schema.json#Pet").optional.desc('description')
    pet:
        - ref.optional.desc('description')
        - http://example.com/schema.json#Pet

Syntax in Python

from validr import T

Welcome = T.dict(
    message='str.desc="Welcome message"'
    message=T.str.desc("Welcome message")
).optional.desc('Welcome Object')

@route('/')
def welcome(
    name: 'str.strip.default="world".desc="Your name"',
    name: T.str.strip.default("world").desc("Your name"),
) -> T.list(Welcome).minlen(3):
    return [{'message': 'hello ' + name}] * 3

enhancement

opened by guyskk 3

Failed to create custom choice validator
I'm trying to make a choice validator like test_custom_validator in test_custom_validator.py, but it will be better with "choices" parameter, so I made this:

def choice_validator(choices: list or tuple): @validator_wrap(string=False) def _validator(value): if value not in choices: raise Invalid('invalid choice') return value return _validator

and used it like:

SP = SchemaParser(validators={'choice': choice_validator}) SP.parse({'type?choice(["A", "B"])': 'blahblah'})

unfortunately this would raise exception:

validr._exception.SchemaError: invalid JSON value in '["A", "B"]'

and I find the parameters between "()" will be split by ",", so there is no way to pass array variable to a custom validator?

It will be great helpful if there is an example of custom validator with parameters.
bug
opened by egoag 3
rename validater to validr
Some changes:

change package name to validr, use import validr instead of import validater

change variables name from validater to validator, include ValidatorString, validr.validators, build_re_validator and builtin_validators
opened by guyskk 3

Error while installing the package

I get this error while installing in an empty environment (virtual environment) with python 3.8.10:

pip install validr
Collecting validr
  Using cached validr-1.2.1.tar.gz (291 kB)
Collecting idna>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting pyparsing>=2.1.0
  Downloading pyparsing-3.0.8-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 154 kB/s 
Collecting terminaltables>=3.1.0
  Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)
Building wheels for collected packages: validr
  Building wheel for validr (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/shayan/dev/test-validr/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mtopyp8x/validr/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mtopyp8x/validr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-kx_fpz1i
       cwd: /tmp/pip-install-mtopyp8x/validr/
  Complete output (7 lines):
  VALIDR_SETUP_MODE=c
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for validr
  Running setup.py clean for validr
Failed to build validr
Installing collected packages: idna, pyparsing, terminaltables, validr
    Running setup.py install for validr ... done
Successfully installed idna-3.3 pyparsing-3.0.8 terminaltables-3.1.10 validr-1.2.1

Also I can't install it in my env created using pipenv with python 3.10.3 and I get:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 41: ordinal not in range(128)

opened by shywn-mrk 2

Error using T.enum (v1.1.3)

python3 --version
Python 3.6.9

pip3 --version
pip 20.0.2 from /home/yumi/.local/lib/python3.6/site-packages/pip (python 3.6)

pip3 install validr
Processing => validr-1.1.3-cp36-cp36m-linux_x86_64.whl

python3
>>> from validr import T
>>> T.enum([1,2,3])
....
  File "/home/yumi/.local/lib/python3.6/site-packages/validr/schema.py", line 424, in _check_items
    raise SchemaError('items must be bool, int, float or str')
validr._exception_c.SchemaError: items must be bool, int, float or str

>>> T.enum(1,2,3)
...
File "/home/matu/.local/lib/python3.6/site-packages/validr/schema.py", line 362, in __call__
    raise SchemaError("can't call with more than one positional argument")
validr._exception_c.SchemaError: can't call with more than one positional argument

opened by elmato 2

Cannot compile for Python 3.10 and 3.11

Steps to reproduce:

$ python3.10 -m venv py310
$ py310/bin/pip install validr

Output:

Installing collected packages: validr
  Running setup.py install for validr ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for validr did not run successfully.
  │ exit code: 1
  ╰─> [1381 lines of output]
      VALIDR_SETUP_MODE=c
      running install
      /home/krat/Projects/py310/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/validr
      copying src/validr/exception.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/schema.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/validator.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/__init__.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/model.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/_validator_py.py -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/_exception_py.py -> build/lib.linux-x86_64-3.10/validr
      creating build/lib.linux-x86_64-3.10/validr/_vendor
      copying src/validr/_vendor/email_validator.py -> build/lib.linux-x86_64-3.10/validr/_vendor
      copying src/validr/_vendor/__init__.py -> build/lib.linux-x86_64-3.10/validr/_vendor
      copying src/validr/_vendor/durationpy.py -> build/lib.linux-x86_64-3.10/validr/_vendor
      copying src/validr/_vendor/fqdn.py -> build/lib.linux-x86_64-3.10/validr/_vendor
      running egg_info
      writing src/validr.egg-info/PKG-INFO
      writing dependency_links to src/validr.egg-info/dependency_links.txt
      writing requirements to src/validr.egg-info/requires.txt
      writing top-level names to src/validr.egg-info/top_level.txt
      reading manifest file 'src/validr.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'src/validr.egg-info/SOURCES.txt'
      copying src/validr/_exception_c.c -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/_exception_c.pyx -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/_validator_c.c -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/_validator_c.pyx -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/model.pyi -> build/lib.linux-x86_64-3.10/validr
      copying src/validr/schema.pyi -> build/lib.linux-x86_64-3.10/validr
      running build_ext
      building 'validr._exception_c' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/src
      creating build/temp.linux-x86_64-3.10/src/validr
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/krat/Projects/py310/include -I/usr/include/python3.10 -c src/validr/_exception_c.c -o build/temp.linux-x86_64-3.10/src/validr/_exception_c.o
      src/validr/_exception_c.c: In function ‘__Pyx_call_return_trace_func’:
      src/validr/_exception_c.c:1075:15: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘use_tracing’; did you mean ‘tracing’?
       1075 |       tstate->use_tracing = 0;
            |               ^~~~~~~~~~~
            |               tracing


      *** a lot of similar lines go here ***


      /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

For Python 3.11 the output differs, but it's still a compilation error.

bug

opened by kr41 4

Bump certifi from 2019.11.28 to 2022.12.7
Bumps certifi from 2019.11.28 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump codecov from 2.0.15 to 2.0.16
Bumps codecov from 2.0.15 to 2.0.16.

Changelog

Sourced from codecov's changelog.

2.0.16

fixed reported command injection vulnerability.

Commits

3a8b06b Version 2.0.16

b2951c0 Merge pull request #231 from codecov/ce-1380

2a80aa4 CE-1380_sanitize_args

73b1b13 Merge pull request #218 from rly/patch-1

80a3fcc Fix broken bitly link in help

ba51a78 Merge pull request #106 from blueyed/verbose

1d5d288 Merge branch 'master' into verbose

3502fdd Merge pull request #201 from takluyver/patch-1

de92d4f Merge pull request #202 from takluyver/patch-2

54033d8 Merge pull request #192 from hugovk/add-3.7

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
uncythonize script

Hi @guyskk

I found your validr_uncython.py scipt quite useful. After modifying it for my needs, I placed it as a standalone project here: https://github.com/JohannesBuchner/uncythonize and uploaded it also to pypi. I am sure I will use it in the future.

Thank you!

opened by JohannesBuchner 1
Feature: use ciso8601 to implement datetime validator

Currently datetime.strptime method is not flexible, it only support vert strict format. Use https://github.com/closeio/ciso8601 is better to parse iso8601 datetime format.

And add tzaware option, if tzaware=True then return datetime object with timezone info (UTC).

The change is compatible, no break changes.
enhancement

opened by guyskk 0