Tomli is a Python library for parsing TOML

Overview

Build Status codecov.io PyPI version

Tomli

A lil' TOML parser

Table of Contents generated with mdformat-toc

Intro

Tomli is a Python library for parsing TOML. Tomli is fully compatible with TOML v1.0.0.

Installation

pip install tomli

Usage

Parse a TOML string

import tomli

toml_str = """
           gretzky = 99

           [kurri]
           jari = 17
           """

toml_dict = tomli.loads(toml_str)
assert toml_dict == {"gretzky": 99, "kurri": {"jari": 17}}

Parse a TOML file

import tomli

with open("path_to_file/conf.toml", encoding="utf-8") as f:
    toml_dict = tomli.load(f)

Handle invalid TOML

import tomli

try:
    toml_dict = tomli.loads("]] this is invalid TOML [[")
except tomli.TOMLDecodeError:
    print("Yep, definitely not valid.")

Note that while the TOMLDecodeError type is public API, error messages of raised instances of it are not. Error messages should not be assumed to stay constant across Tomli versions.

Construct decimal.Decimals from TOML floats

from decimal import Decimal
import tomli

toml_dict = tomli.loads("precision-matters = 0.982492", parse_float=Decimal)
assert isinstance(toml_dict["precision-matters"], Decimal)

Note that you may replace decimal.Decimal with any callable that converts a TOML float from string to any Python type (except list or dict). The decimal.Decimal type is, however, the most typical replacement when float inaccuracies can not be tolerated.

FAQ

Why this parser?

  • it's lil'
  • pure Python with zero dependencies
  • the fastest pure Python parser *: 14x as fast as tomlkit, 2.4x as fast as toml
  • outputs basic data types only
  • 100% spec compliant: passes all tests in a test set soon to be merged to the official compliance tests for TOML repository
  • thoroughly tested: 100% branch coverage

Is comment preserving round-trip parsing supported?

No.

The tomli.loads function returns a plain dict that is populated with builtin types and types from the standard library only. Preserving comments requires a custom type to be returned so will not be supported, at least not by the tomli.loads and tomli.load functions.

Look into TOML Kit if preservation of style is what you need.

Is there a dumps, write or encode function?

Tomli-W is the write-only counterpart of Tomli, providing dump and dumps functions.

The core library does not include write capability, as most TOML use cases are read-only, and Tomli intends to be minimal.

How do TOML types map into Python types?

TOML type Python type Details
Document Root dict
Key str
String str
Integer int
Float float
Boolean bool
Offset Date-Time datetime.datetime tzinfo attribute set to an instance of datetime.timezone
Local Date-Time datetime.datetime tzinfo attribute set to None
Local Date datetime.date
Local Time datetime.time
Array list
Table dict
Inline Table dict

Performance

The benchmark/ folder in this repository contains a performance benchmark for comparing the various Python TOML parsers. The benchmark can be run with tox -e benchmark-pypi. Running the benchmark on my personal computer output the following:

foo@bar:~/dev/tomli$ tox -e benchmark-pypi
benchmark-pypi installed: attrs==19.3.0,click==7.1.2,pytomlpp==1.0.2,qtoml==0.3.0,rtoml==0.7.0,toml==0.10.2,tomli==1.0.4,tomlkit==0.7.2
benchmark-pypi run-test-pre: PYTHONHASHSEED='2995948371'
benchmark-pypi run-test: commands[0] | python -c 'import datetime; print(datetime.date.today())'
2021-07-05
benchmark-pypi run-test: commands[1] | python --version
Python 3.8.5
benchmark-pypi run-test: commands[2] | python benchmark/run.py
Parsing data.toml 5000 times:
------------------------------------------------------
    parser |  exec time | performance (more is better)
-----------+------------+-----------------------------
     rtoml |    0.902 s | baseline (100%)
  pytomlpp |     1.09 s | 83.00%
     tomli |     3.82 s | 23.60%
      toml |     9.27 s | 9.73%
     qtoml |     11.5 s | 7.84%
   tomlkit |     55.1 s | 1.64%

The parsers are ordered from fastest to slowest, using the fastest parser as baseline. Tomli performed the best out of all pure Python TOML parsers, losing only to pytomlpp (wraps C++) and rtoml (wraps Rust).

Comments
  • Please consider pushing tomli into stdlib

    Please consider pushing tomli into stdlib

    Long story short, having a TOML parser in Python stdlib would be great and tomli seems to be the best implementation available right now, so also the best candidate for stdlib. Would you be interested in trying to push it?

    The relevant CPython bug is: https://bugs.python.org/issue40059

    question 
    opened by mgorny 48
  • tomli violates PEP517 Build Requirements

    tomli violates PEP517 Build Requirements

    The PEP517 Build Requirements specifically disallow dependency cycles:

    • Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.

    • Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds. However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.

    question 
    opened by jameshilliard 19
  • Please include tests in the pypi tarball

    Please include tests in the pypi tarball

    On Gentoo we really want to let our users run tests before installing the package. Normally, we just use the autogenerated GitHub archives for that. However, since tomli uses pyproject.toml, this will effectively create a cyclic dependency between flit and itself once flit switches over from toml (and it already creates one for the Gentoo build tool).

    Could you please include the necessary test files in the generated pypi tarballs? Since they include a backwards compatible setup.py script, they would really help us avoid cyclic dependencies.

    question wontfix 
    opened by mgorny 16
  • Support reading TOML from a filename.

    Support reading TOML from a filename.

    Both uiri/toml and samuelcolvin/rtoml support filenames in their load functions. They are then responsible for opening the file with the correct encoding and reading its contents, before calling loads.

    However, tomli only supports file-like objects, leaving it to the user to open the file and, inevitably, forgetting to use UTF-8 as the encoding (e.g. https://github.com/pulls?q=is%3Apr+author%3Adomdfcoding+UTF-8).

    I propose copying toml and rtoml and adding support for strings and os.PathLike objects. That way the user can write:

    config = tomli.load("pyproject.toml")
    

    or

    config = tomli.load(pathlib.Path("pyproject.toml"))
    

    without having to worry about the encoding, and saving a line over the with open approach.


    What do you think? If you're happy with the idea I'll submit a PR

    enhancement 
    opened by domdfcoding 14
  • Slightly speed up execution time of common cases

    Slightly speed up execution time of common cases

    I'm wrapping up re-writing a CLI tool where responsiveness is critical and during the switch from toml to tomli I noticed load time increased by ~1-2 milliseconds. This PR brings it down the equivalent amount.

    If you don't like the singleton you can eventually use https://www.python.org/dev/peps/pep-0562/ but I haven't benchmarked that way.

    opened by ofek 10
  • Please allow flit to synthesize setup.py for this package

    Please allow flit to synthesize setup.py for this package

    As part of bootstrapping base pep517 stack you now have distutils -> setuptools -> tomli + pep517 + build. Since you are apparently creating tarballs with --no-setup-py, this bootstrap chain is now impossible which makes pep517 a misery for distros (see https://github.com/pypa/pep517/pull/125). Can you please consider not using this switch?

    question 
    opened by nanonyme 10
  • Compile code with Mypyc

    Compile code with Mypyc

    Hello again! Let me know what you think πŸ˜„

    This PR compiles code with Mypyc. To do so, I've switched the build system to hatchling in order to use the hatch-mypyc build hook plugin. Hatchling is the minimal build backend of Hatch (v1 rewrite of CLI being released next week). You can think of it as the same distinction between flit-core & flit or poetry-core & poetry, with hatchling being more similar to flit-core in that only accepted standards are used e.g. project metadata format (PEP 621), dependency specification (PEP 631), etc.

    I added binary wheels to the release workflow using the now ubiquitous cibuildwheel. Here is an example of the files produced: https://pypi.org/project/hatch-showcase/#files

    The only thing I couldn't preserve was the hack where we change the __module__ of TOMLDecodeError.

    On my Windows machine, tox -e benchmark shows a 2-3x performance improvement.

    opened by ofek 8
  • Include setup.py in sdist

    Include setup.py in sdist

    I noticed setup.py was removed here, however since tomli is a required dependency for bootstrapping flit this should be included to make installing flit easier for distros that need it for setting up a PEP-517 build environment.

    duplicate question 
    opened by jameshilliard 7
  • Type stub incompatiblity with tarfile.ExFileObject (i.e. tarfile.TarFile.extractfile() result)

    Type stub incompatiblity with tarfile.ExFileObject (i.e. tarfile.TarFile.extractfile() result)

    Please consider the following snippet:

    import tomli
    import tarfile
    
    with tarfile.open("test.tar") as tar:
        f = tar.extractfile("test.toml")
        assert f is not None
        tomli.load(f)
    

    When I run mypy on it, I get:

    test.py:7: error: Argument 1 to "load" has incompatible type "IO[bytes]"; expected "BinaryIO"  [arg-type]
    Found 1 error in 1 file (checked 1 source file)
    

    To be honest, typing docs don't really say what the difference between IO[bytes] and BinaryIO is. Besides type stubs, the code works. FWICS, tomllib would suffer from the same problem if not for the fact that mypy overrides its type stub to:

    def load(__fp: SupportsRead[bytes], *, parse_float: Callable[[str], Any] = ...) -> dict[str, Any]: ...
    
    opened by mgorny 6
  • Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0`

    Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0`

    I can not download tomli because there is a cyclic dependency with flit_core.

    Reproducer:

    python3.9 -m venv test
    ./test/bin/pip download --no-binary=":all:"  tomli  # fails
    ./test/bin/pip download --no-binary=":all:" flit_core # fails
    

    Example output:

    Collecting tomli
      File was already downloaded XXX/tomli-1.2.1.tar.gz
      Installing build dependencies ... error
      ERROR: Command errored out with exit status 2:
       command: XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
           cwd: None
      Complete output (31 lines):
      Collecting flit_core<4,>=3.2.0
        Downloading flit_core-3.4.0.tar.gz (27 kB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
          Preparing wheel metadata: started
          Preparing wheel metadata: finished with status 'done'
      Collecting tomli
        Downloading tomli-1.2.1.tar.gz (14 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
          status = self.run(options, args)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 357, in run
          resolver.resolve(requirement_set)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
          discovered_reqs.extend(self._resolve_one(requirement_set, req))
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
          abstract_dist = self._get_abstract_dist_for(req_to_install)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
          abstract_dist = self.preparer.prepare_linked_requirement(req)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
          abstract_dist = _get_prepared_distribution(
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
          with req_tracker.track(req):
        File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
          return next(self.gen)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
          self.add(req)
        File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
      ----------------------------------------
    ERROR: Command errored out with exit status 2: XXX/test/bin/python3.9 XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.
    
    question 
    opened by lummax 6
  • Why splitting the library into two?

    Why splitting the library into two?

    Hi! I like the idea of having a simple pure Python library for TOML! And I understand the motivation of keeping it minimal, however, I am not sure if splitting the package into two is the optimal approach. I would like to share my thoughts on that.

    1. If I am a user that does NOT need dump(s), what is the benefit of NOT having dump(s)? By looking at tomli-w, it seems that the benefit is the absence of ~150 lines of Python code. If it is true, I personally do not see use cases where this benefit can be important. I understand the goal of "theoretically minimal library". Are you sure that this goal is practical?
    2. If I am a user that needs dump(s), I have to install two packages instead of one and import two different packages instead of one. This setup does not feel minimal to me (unfortunately, this is exactly my case).
    3. In fact, tomli is not minimal: loads can be replaced by using StringIO and load :) While this is rather a joke, I would like to say that the "minimalism", when not absolute, becomes a subjective thing.

    Taking all this into account, merging the two libraries into one looks good to me. What do you think?

    question 
    opened by Yura52 6
  • Do not use recursion

    Do not use recursion

    I noticed that there are couple of TOML test files without JSON counterparts (https://github.com/hukkin/tomli/blob/master/tests/data/toml-lang-compliance/valid/qa/qa-array-inline-nested-1000.toml, https://github.com/hukkin/tomli/blob/master/tests/data/toml-lang-compliance/valid/qa/qa-table-inline-nested-1000.toml) that couldn't be loaded with the current version due to deep recursion. Maybe they're not very realistic use cases, but the fix isn't that complicated. On my computer there's a 1-2% performance hit, but you might want to test more. One another thing is that I don't have deep knowledge about NestedDict, Flags and other more complicated inner workings of the software. I just naively copied more or less everything to the stack even though they might be available in some smarter way.

    opened by oittaa 2
Owner
Taneli Hukkinen
Taneli Hukkinen
AWS Lambda - Parsing Cloudwatch Data and sending the response via email.

AWS Lambda - Parsing Cloudwatch Data and sending the response via email. Author: Evan Erickson Language: Python Backend: AWS / Serverless / AWS Lambda

Evan Scott Erickson 1 Nov 14, 2021
Aio-binance-library - Async library for connecting to the Binance API on Python

aio-binance-library Async library for connecting to the Binance API on Python Th

GRinvest 10 Nov 21, 2022
alpaca-trade-api-python is a python library for the Alpaca Commission Free Trading API.

alpaca-trade-api-python is a python library for the Alpaca Commission Free Trading API. It allows rapid trading algo development easily, with support for both REST and streaming data interfaces

Alpaca 1.5k Jan 9, 2023
🐍 The official Python client library for Google's discovery based APIs.

Google API Client This is the Python client library for Google's discovery based APIs. To get started, please see the docs folder. These client librar

Google APIs 6.2k Jan 8, 2023
Python bindings for ArrayFire: A general purpose GPU library.

ArrayFire Python Bindings ArrayFire is a high performance library for parallel computing with an easy-to-use API. It enables users to write scientific

ArrayFire 402 Dec 20, 2022
Balanced API library in python.

Balanced Online Marketplace Payments v1.x requires Balanced API 1.1. Use v0.x for Balanced API 1.0. Installation pip install balanced Usage View Bala

Balanced 70 Oct 4, 2022
python library to the bitly api

bitly API python library Installation pip install bitly_api Run tests Your username is the lowercase name shown when you login to bitly, your access

Bitly 245 Aug 14, 2022
Braintree Python library

Braintree Python library The Braintree Python library provides integration access to the Braintree Gateway. TLS 1.2 required The Payment Card Industry

Braintree 230 Dec 18, 2022
A Python library for the Buildkite API

PyBuildkite A Python library and client for the Buildkite API. Usage To get the package, execute: pip install pybuildkite Then set up an instance of

Peter Yasi 29 Nov 30, 2022
Unofficial Coinbase Python Library

Unofficial Coinbase Python Library Python Library for the Coinbase API for use with three legged oAuth2 and classic API key usage Version 0.3.0 Requir

George Sibble 104 Dec 1, 2022
A Python library for the Discourse API

pydiscourse A Python library for working with Discourse. This is a fork of the original Tindie version. It was forked to include fixes, additional fun

Ben Lopatin 72 Oct 14, 2022
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 6.1k Jan 3, 2023
A Python Library to interface with Flickr REST API, OAuth & JSON Responses

Python-Flickr Python-Flickr is A Python library to interface with Flickr REST API & OAuth Features Photo Uploading Retrieve user information Common Fl

Mike Helmick 40 Sep 25, 2021
Unofficial GoPro API Library for Python - connect to GoPro via WiFi.

GoPro API for Python Unofficial GoPro API Library for Python - connect to GoPro cameras via WiFi. Compatibility: HERO3 HERO3+ HERO4 (including HERO Se

Konrad Iturbe 1.3k Jan 1, 2023
A Python library wrapping the iFixit (Dozuki) API.

A Python library wrapping the iFixit API. Status Working, but incomplete. Fully tested and documented. Hacking I highly recommend using virtualenv: [$

James Pearson Hughes 13 May 24, 2021
:snake: A simple library to fetch data from the iTunes Store API made for Python >= 3.5

itunespy itunespy is a simple library to fetch data from the iTunes Store API made for Python 3.5 and beyond. Important: Since version 1.6 itunespy no

Fran GonzΓ‘lez 56 Dec 22, 2022
Python JIRA Library is the easiest way to automate JIRA. Support for py27 was dropped on 2019-10-14, do not raise bugs related to it.

Jira Python Library This library eases the use of the Jira REST API from Python and it has been used in production for years. As this is an open-sourc

PyContribs 1.7k Jan 6, 2023
The official Python client library for the Kite Connect trading APIs

The Kite Connect API Python client - v3 The official Python client for communicating with the Kite Connect API. Kite Connect is a set of REST-like API

Zerodha Technology 756 Jan 6, 2023
A Python Library to interface with LinkedIn API, OAuth and JSON responses

#Overview Here's another library based on the LinkedIn API, OAuth and JSON responses. Hope this documentation explains everything you need to get star

Mike Helmick 69 Dec 11, 2022