According to the optional keys docs and the sources there is no support for an implicit default value for an optional key. My use case: I potentially have to configure a very long list of key value mappings for my command line application. The majority of the optional-keys usually have a default value which is implicit but from application/user perspective reasonable, not surprising. I know implicit data may be bad. Anyway... what do you think?

- mandatory-key: ...
  optional-key:
- mandatory-key: ...
  optional-key:
(... potentially very long list of other mappings ...)

Hi, i found a bug when trying to "or" deeper nested structures:

from strictyaml import load, Map, MapPattern, Str, Int

yaml="""
configuration: 
  test1:
    helptext: "some text"
  test2:
    helptext: "some other text."
    min: 0
"""

schema = Map(
    {
        "configuration": MapPattern(
            Str(), 
            Map({
                "helptext": Str(), 
            }) | Map({
                "helptext": Str(), 
                "min": Int(), 
            }),
            minimum_keys=1
        ),
    }
)

data = load(yaml, schema=schema)
print(data.data)

which results in

strictyaml.exceptions.YAMLValidationError: when expecting an integer
found arbitrary text
  in "<unicode string>", line 5, column 1:
        helptext: some other text.
    ^ (line: 5)

the strange thing is, it "works" when themin validator is a Str() so i guess the validators are applied to the wrong content?

Consider this YAML:

array:
  - string 1!
  - string: 2?

Which parses into this:

{'array': ['string 1!', {'string': '2?'}]}

The value determines its type. You've eliminated this behavior for primitive types:

python: 3.5.3
postgres: 9.3

Both are strings. But there is still implicit typing between string and map.

An array element syntax like:

array:
  -
  key1: value1
  key2: value2
  -
  key1: value1
  key2: value2
  - some: text

would fix this behavior but then it wouldn't be a subset of YAML anymore 😄

This is a replacement for https://github.com/crdoconnor/strictyaml/pull/75

I merged and then reverted the original PR. This unmerged branch contains a revert of that revert.

This code unfortunately causes story "Boolean (Bool)/Update boolean values with string and bool type" to fail (and possibly some others). @wwoods can you run the test and see what happened?

Once all the tests pass - i.e. "hk regression" has no failures, I'll be happy to merge a new pull request.

Thanks for the great library! I'm hoping to use this instead of pyyaml when parsing OpenAPI 3.0.x documents, but I'm running into an issue with the Security Requirements Object of a path item.

From that link, emphasis mine:

Each name MUST correspond to a security scheme which is declared in the Security Schemes under the Components Object. If the security scheme is of type "oauth2" or "openIdConnect", then the value is a list of scope names required for the execution. For other security scheme types, the array MUST be empty.

This means I've got paths like this:

paths:
  /users/{userId}/widgets/{widgetId}/revisions:
    post:
      operationId: createWidgetRevision
      security:
        - apiToken: []

Which fails as expected with FLowMappingDisallowed:

$ ipython
Python 3.6.5 (default, Apr  1 2018, 15:30:28) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from strictyaml import load

In [2]: doc="""paths:
   ...:   /users/{userId}/widgets/{widgetId}/revisions:
   ...:     post:
   ...:       operationId: createWidgetRevision
   ...:       security:
   ...:         - apiToken: []"""

In [3]: load(doc)
FlowMappingDisallowed [traceback clipped]

Any recommendations? There's also a commonly-used form for an unauthenticated method with a similar issue:

paths:
  /items/{itemId}:
    put:
      operationId: updateItem
      security:
        - {}

#62 mentions that None is not supported as default value in Optional. Fix this by adding a unique object and checking for identity when determining if the default value exists or not.

Also add the default value to __repr__

Btw: is there a timeline for when default values in Optionals will become stable? They haven't changed in the last two years.

I was trying to find out on the project site whether specifying a schema prevents the document from having additional fields.

In other words, if a document's schema is a superset of the expected schema, is it considered valid?

Before this commit, schemas could be violated when assigning to Map/Sequence members. Now, modifications to the data must fit the data's schema.

Furthermore, if the node on which setitem is called has a compound schema, the selected validator within the compound schema may change correctly.

This fixes #72, #76, and supersedes #75. @crdoconnor I'm pretty happy with this in that it works on all tests without any changes, and supports the new test suite. There are two strange aspects which you may want to weigh in on:

1. The _strictparsed option, which to me had no clear semantics, is overwritten with self. This works to preserve the hierarchy and required methods, but I'm not sure if this would break anything else. At any rate, again, all the tests pass, and it seems to behave sanely.

2. I make no claims about optimality in terms of speed of this approach. Frankly, you'd probably need to rewrite a few of the internals if that were the chief concern of this library. That said, I did try to prevent multiple validations a few places, but there are likely more ways to invoke validations of already-validated content, which is harmless but could eventually lead to a performance problem.

Please let me know if you do/do not merge this; I'm going ahead with my professional project assuming this (or a slight variant) will be accepted and rolled into a version that I'll be able to have clients pull down in the near future.

• Feature request

How many indent spaces should I use?

I encounter many YAML files, some indent 4 spaces, some indent 2 spaces, and some indent 2 spaces before - list, and others not. Sometimes I got confused when they mixed:

example:
  indent: 2 spaces
  items:
  - one
  - two
  others:
    - three
    - four
  inner:
      indent: 4 spaces
      items:
      - five
      - six
      others:
          - seven
          - eight

strictyaml does schema based validation and reports exact text positions of erroneous input. However in addition to schema validation, the application may perform other checks which are impossible to express in terms of schema. It would be nice to be able to report to user exact text position of offending data. I was able to come up with a quick-and-dirty proof of concept (tested only on python 3.5) that uses type(name, bases, dict) to construct subclasses of built-in types (dict, list, int, str) that contain start_mark and end_mark of nodes producing the corresponding data. I create those sub-classed instances in construct_object. I can send you the above-mentioned prototype should you find it useful.

Consider the following: https://github.com/crdoconnor/strictyaml/blob/master/strictyaml/exceptions.py#L4

StrictYAMLError should not inherit from YAMLError, allowing people to make a clear distinction between errors that are raised by your library and errors that are raised by the underlying one.

I know one could simply do:

try:
  # ...
except YAMLError:
  pass
except StrictYAMLError:
  pass

But (1) this implies that YAMLError is in the current scope (it is not even exposed by your library, but that's not the point), and (2) YAMLError are caught before StrictYAMLError.

I used version: 1.6.1. The following program is correct:

s = "mode: default"

schema_map = {
    "mode": sy.Str(),
    sy.Optional("duration", default=None, drop_if_none=False): sy.EmptyNone()
    | sy.Float(),
}

config_yaml = sy.load(
    yaml_string=s,
    schema=sy.Map(schema_map),
)

But when I change the type order of optional duration like following:

s = "mode: default"

schema_map = {
    "mode": sy.Str(),
    sy.Optional("duration", default=None, drop_if_none=False): sy.Float()
    | sy.EmptyNone(),
}

config_yaml = sy.load(
    yaml_string=s,
    schema=sy.Map(schema_map),
)

then there would be an error, here is the error log:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../vendor/lib/python3.10/site-packages/strictyaml/parser.py:323: in load
    return generic_load(yaml_string, schema=schema, label=label)
../vendor/lib/python3.10/site-packages/strictyaml/parser.py:301: in generic_load
    return schema(YAMLChunk(document, label=label))
../vendor/lib/python3.10/site-packages/strictyaml/validators.py:17: in __call__
    self.validate(chunk)
../vendor/lib/python3.10/site-packages/strictyaml/compound.py:180: in validate
    new_value = value_validator(
../vendor/lib/python3.10/site-packages/strictyaml/validators.py:107: in __call__
    result = self._validator_a(chunk)
../vendor/lib/python3.10/site-packages/strictyaml/scalar.py:27: in __call__
    return YAML(chunk, validator=self)
../vendor/lib/python3.10/site-packages/strictyaml/representation.py:63: in __init__
    self._value = validator.validate(value)
../vendor/lib/python3.10/site-packages/strictyaml/scalar.py:30: in validate
    return self.validate_scalar(chunk)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Float()
chunk = <strictyaml.yamllocation.YAMLChunk object at 0x7f64f2212aa0>

    def validate_scalar(self, chunk):
        val = chunk.contents
        if utils.is_infinity(val) or utils.is_not_a_number(val):
            val = val.replace(".", "")
        elif not utils.is_decimal(val):
            chunk.expecting_but_found("when expecting a float")
        # Only Python 3.6+ supports underscores in numeric literals
>       return float(val.replace("_", ""))
E       ValueError: could not convert string to float: ''

../vendor/lib/python3.10/site-packages/strictyaml/scalar.py:230: ValueError

But if changing sy.Float() to sy.Int(), this error would not arise.

I got started on moving to this, but now I discover your claim is not at all true - strictyaml does not offer even one function that is the same as pyyaml.

The one facility that the two appear to share is load, except good pyyaml code is using safe_load - but in fact load is a "false friend", as I discovered when I put it in my code:

>>> strictyaml.load('1') + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'YAML' and 'int'

Yes, I understand why you are doing this, so you can preserve comments, but why not use another name considering this is an entirely different software service that returns a radically different type?

And there isn't a dump or a dumps like JSON and pretty well all other such systems have. Are we instead supposed to do as_document({"x": 1}).as_yaml()?

How much work would it really have taken to create a load, dump/dumps so you'd be compatible with pyyaml, json, toml, and pretty well all other serialization tools in Python?

Time is all humans have in their lives, and I just wasted 40 minutes of it because your description is the reverse of the truth.

Your website includes a lot of criticism of YAML’s Norway problem (where NO would parse as the boolean false), but it neglects to mention that this behavior was already removed thirteen years ago in YAML 1.2, and even includes this incorrect statement:

The most tragic aspect of this bug, however, is that it is intended behavior according to the YAML 1.2 specification.

It was intended according to the YAML 1.1 specification, but in YAML 1.2, the only recognized booleans are true and false.

Now, there are of course still arguments to be made—for example, some popular libraries still only support YAML 1.1. But these arguments should be made clearly and fairly.

Incomplete revalidation also causes a mismatch between the ruamel and strictyaml internal states in the YAMLChunk object. This is the same outcome as in #183, but the initial cause of the inconsistency is different, so I've listed this as a separate issue.

An example of the error:

from strictyaml import Int, Map, Str, as_document

dat = {"a": 1, "b": "x"}
schema1 = Map({"a": Int(), "b": Int()})
schema2 = Map({"a": Int(), "b": Str()})

# The state is consistent here
yml._chunk.contents
# ordereddict([('a', '1'), ('b', 'x')])
yml._chunk.strictparsed()
# ordereddict([(YAML(a), YAML(1)), (YAML(b), YAML(x))])

yml.revalidate(schema1)
# Fails (as it should)

# But the state is now inconsistent
yml._chunk.contents
# ordereddict([('a', '1'), ('b', 'x')])
yml._chunk.strictparsed()
# ordereddict([(YAML(b), YAML(x)), (YAML(a), YAML(1))])

The mismatch in the internal state means that revalidation with a correct schema will fail:

yml.revalidate(schema2)

...
strictyaml.exceptions.YAMLValidationError: when expecting an integer
found arbitrary text
  in "<unicode string>", line 2, column 1:
    b: x
    ^ (line: 2)

I've found some weird bug: calling revalidate() for the second time raises an Invalid state exception when optional fields with default values are present in the schema, but not in the YAML document. Here's a snippet to reproduce the problem.

from strictyaml import load, Map, Int, Optional

doc = "x: 1"

# This works fine
schema = Map({ "x": Int(), Optional("y"): Int() })
yaml = load(doc, schema)
try:
    for i in range(1,6):
        yaml.revalidate(schema)
        print(i, end=" ")
except Exception as e:
    print(e)
print("")

# This raises an exception on the second iteration
schema = Map({ "x": Int(), Optional("y", default=42): Int() })
yaml = load(doc, schema)
try:
    for i in range(1,6):
        yaml.revalidate(schema)
        print(i, end=" ")
except Exception as e:
    print(e)

The generated output:

1 2 3 4 5 
1 Invalid state

It seems that the type of the field does not matter: I've got the same behavior for Str and Any.

On the other hand, the exception is not raised when the YAML object is modified, either by removing the optional value or modifying another one i.e. no exception is raised by the following loops

schema = Map({ "x": Int(), Optional("y", default=42): Int(), Optional("z", default=42): Int() })
yaml = load(doc, schema)
try:
    for i in range(1,6):
        yaml['y'] = 18
        del yaml['y']
        yaml.revalidate(schema)

    for i in range(1,6):
        yaml['x'] = i
        yaml.revalidate(schema)

except Exception as e:
    print(e)

It looks to me that revalidate() uses some internal values that are reset whenever the YAML object is modified, but not otherwise.