Long story short, having a TOML parser in Python stdlib would be great and tomli seems to be the best implementation available right now, so also the best candidate for stdlib. Would you be interested in trying to push it?

The relevant CPython bug is: https://bugs.python.org/issue40059

The PEP517 Build Requirements specifically disallow dependency cycles:

• Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.

• Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds. However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.

On Gentoo we really want to let our users run tests before installing the package. Normally, we just use the autogenerated GitHub archives for that. However, since tomli uses pyproject.toml, this will effectively create a cyclic dependency between flit and itself once flit switches over from toml (and it already creates one for the Gentoo build tool).

Could you please include the necessary test files in the generated pypi tarballs? Since they include a backwards compatible setup.py script, they would really help us avoid cyclic dependencies.

Both uiri/toml and samuelcolvin/rtoml support filenames in their load functions. They are then responsible for opening the file with the correct encoding and reading its contents, before calling loads.

However, tomli only supports file-like objects, leaving it to the user to open the file and, inevitably, forgetting to use UTF-8 as the encoding (e.g. https://github.com/pulls?q=is%3Apr+author%3Adomdfcoding+UTF-8).

I propose copying toml and rtoml and adding support for strings and os.PathLike objects. That way the user can write:

config = tomli.load("pyproject.toml")

or

config = tomli.load(pathlib.Path("pyproject.toml"))

without having to worry about the encoding, and saving a line over the with open approach.

What do you think? If you're happy with the idea I'll submit a PR

I'm wrapping up re-writing a CLI tool where responsiveness is critical and during the switch from toml to tomli I noticed load time increased by ~1-2 milliseconds. This PR brings it down the equivalent amount.

If you don't like the singleton you can eventually use https://www.python.org/dev/peps/pep-0562/ but I haven't benchmarked that way.

As part of bootstrapping base pep517 stack you now have distutils -> setuptools -> tomli + pep517 + build. Since you are apparently creating tarballs with --no-setup-py, this bootstrap chain is now impossible which makes pep517 a misery for distros (see https://github.com/pypa/pep517/pull/125). Can you please consider not using this switch?

Hello again! Let me know what you think 😄

This PR compiles code with Mypyc. To do so, I've switched the build system to hatchling in order to use the hatch-mypyc build hook plugin. Hatchling is the minimal build backend of Hatch (v1 rewrite of CLI being released next week). You can think of it as the same distinction between flit-core & flit or poetry-core & poetry, with hatchling being more similar to flit-core in that only accepted standards are used e.g. project metadata format (PEP 621), dependency specification (PEP 631), etc.

I added binary wheels to the release workflow using the now ubiquitous cibuildwheel. Here is an example of the files produced: https://pypi.org/project/hatch-showcase/#files

The only thing I couldn't preserve was the hack where we change the __module__ of TOMLDecodeError.

On my Windows machine, tox -e benchmark shows a 2-3x performance improvement.

I noticed setup.py was removed here, however since tomli is a required dependency for bootstrapping flit this should be included to make installing flit easier for distros that need it for setting up a PEP-517 build environment.

Please consider the following snippet:

import tomli
import tarfile

with tarfile.open("test.tar") as tar:
    f = tar.extractfile("test.toml")
    assert f is not None
    tomli.load(f)

When I run mypy on it, I get:

test.py:7: error: Argument 1 to "load" has incompatible type "IO[bytes]"; expected "BinaryIO"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

To be honest, typing docs don't really say what the difference between IO[bytes] and BinaryIO is. Besides type stubs, the code works. FWICS, tomllib would suffer from the same problem if not for the fact that mypy overrides its type stub to:

def load(__fp: SupportsRead[bytes], *, parse_float: Callable[[str], Any] = ...) -> dict[str, Any]: ...

I can not download tomli because there is a cyclic dependency with flit_core.

Reproducer:

python3.9 -m venv test
./test/bin/pip download --no-binary=":all:"  tomli  # fails
./test/bin/pip download --no-binary=":all:" flit_core # fails

Example output:

Collecting tomli
  File was already downloaded XXX/tomli-1.2.1.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
       cwd: None
  Complete output (31 lines):
  Collecting flit_core<4,>=3.2.0
    Downloading flit_core-3.4.0.tar.gz (27 kB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting tomli
    Downloading tomli-1.2.1.tar.gz (14 kB)
  ERROR: Exception:
  Traceback (most recent call last):
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
      status = self.run(options, args)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 357, in run
      resolver.resolve(requirement_set)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
      abstract_dist = _get_prepared_distribution(
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
      with req_tracker.track(req):
    File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
      return next(self.gen)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
      self.add(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
      raise LookupError(message)
  LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
  ----------------------------------------
ERROR: Command errored out with exit status 2: XXX/test/bin/python3.9 XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.

Hi! I like the idea of having a simple pure Python library for TOML! And I understand the motivation of keeping it minimal, however, I am not sure if splitting the package into two is the optimal approach. I would like to share my thoughts on that.

1. If I am a user that does NOT need dump(s), what is the benefit of NOT having dump(s)? By looking at tomli-w, it seems that the benefit is the absence of ~150 lines of Python code. If it is true, I personally do not see use cases where this benefit can be important. I understand the goal of "theoretically minimal library". Are you sure that this goal is practical?
2. If I am a user that needs dump(s), I have to install two packages instead of one and import two different packages instead of one. This setup does not feel minimal to me (unfortunately, this is exactly my case).
3. In fact, tomli is not minimal: loads can be replaced by using StringIO and load :) While this is rather a joke, I would like to say that the "minimalism", when not absolute, becomes a subjective thing.

Taking all this into account, merging the two libraries into one looks good to me. What do you think?

updates:

• github.com/PyCQA/isort: c5e8fa75dda5f764d20f66a215d71c21cfa198e1 → 4e97b170469b7c8ef29afe944ebfb057791457aa
• github.com/psf/black: 27d20144a7517594e24a1649451177b2a11284be → 2ddea293a88919650266472186620a98a4a8bb37
• https://github.com/myint/docformatter → https://github.com/PyCQA/docformatter
• github.com/PyCQA/docformatter: 8b8dae1d990e2e4fec7ff6704a79c23a403cd605 → 11d9a03865ae1f859d56517daec72e4f9f2401f0
• github.com/pre-commit/pre-commit-hooks: 3298ddab3c13dd77d6ce1fc0baf97691430d84b0 → f71fa2c1f9cf5cb705f73dffe4b21f7c61470ba9
• github.com/PyCQA/flake8: f8e1b317742036ff11ff86356fd2b68147e169f7 → c838a5e98878f17889cfce311e1406d252f87ec5
• github.com/pre-commit/mirrors-mypy: 33f4a30be4e66513c51a857712fc732e1a9ddd78 → 0b037c2b59aa62dc3be3287d175295f1a5547eb9

I noticed that there are couple of TOML test files without JSON counterparts (https://github.com/hukkin/tomli/blob/master/tests/data/toml-lang-compliance/valid/qa/qa-array-inline-nested-1000.toml, https://github.com/hukkin/tomli/blob/master/tests/data/toml-lang-compliance/valid/qa/qa-table-inline-nested-1000.toml) that couldn't be loaded with the current version due to deep recursion. Maybe they're not very realistic use cases, but the fix isn't that complicated. On my computer there's a 1-2% performance hit, but you might want to test more. One another thing is that I don't have deep knowledge about NestedDict, Flags and other more complicated inner workings of the software. I just naively copied more or less everything to the stack even though they might be available in some smarter way.