hukkin / tomli Goto Github PK
View Code? Open in Web Editor NEWA lil' TOML parser
License: MIT License
A lil' TOML parser
License: MIT License
Hello, I'm attempting to write forward-looking code which uses the documented tomli
/tomllib
compatibility layer:
try:
import tomllib
except ModuleNotFoundError:
import tomli as tomllib
result = tomllib.loads("")
reveal_type(result)
When I check this code with mypy, I am met with the following errors:
(Python 3.10.9
, mypy 0.991
)
$ mypy example.py
example.py:2: error: Cannot find implementation or library stub for module named "tomllib" [import]
example.py:2: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
example.py:4: error: Name "tomllib" already defined (possibly by an import) [no-redef]
example.py:7: note: Revealed type is "Any"
Found 2 errors in 1 file (checked 1 source file
Line 2: The tomllib
import error is expected since I am running Python 3.10 (and it was added to the standard lib in 3.11). This is resolved by ignoring the import error.
Line 4: The Name "tomllib" already defined
error is resolved by upgrading to mypy 1.2.0
or ingoring the redef error.
try:
import tomllib # type: ignore[import]
except ModuleNotFoundError:
import tomli as tomllib # type: ignore[no-redef]
result = tomllib.loads("")
reveal_type(result)
$ mypy example.py
example.py:7: note: Revealed type is "Any"
Success: no issues found in 1 source file
tomllib
is Any
, so all functions and their return values are Any
as well.When the compatibility layer is not used, the typing is correctly provided (presumably via typeshed, although I can't find it?)
import tomli
reveal_type(tomli)
result = tomli.loads("")
reveal_type(result)
$ mypy example_nolayer.py
example_nolayer.py:2: note: Revealed type is "types.ModuleType"
example_nolayer.py:5: note: Revealed type is "builtins.dict[builtins.str, Any]"
Is this a limitation of mypy that we have to live with, or can the pattern be somehow updated to resolve these issues?
datetime
in standard library has a microsecond precision, but for example numpy.datetime64
has a nanosecond precision. It would be pretty easy to add another parameter parse_datetime
like the current parse_float
. Even something a bit more complex like the custom JSON encoders and decoders might be useful. Or do you think custom parsers would complicate things too much?
On Gentoo we really want to let our users run tests before installing the package. Normally, we just use the autogenerated GitHub archives for that. However, since tomli uses pyproject.toml
, this will effectively create a cyclic dependency between flit and itself once flit switches over from toml (and it already creates one for the Gentoo build tool).
Could you please include the necessary test files in the generated pypi tarballs? Since they include a backwards compatible setup.py
script, they would really help us avoid cyclic dependencies.
The PEP517 Build Requirements specifically disallow dependency cycles:
Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.
Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds. However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.
I've made a fork of tomli that supports micropython:
https://github.com/BrianPugh/micropython-tomli
A lot of the changes make the codebase worse from a cpython perspective (removed type hints, uglier regex, etc etc), so I'm not looking to make a PR or anything. I was wondering if you'd be interested in linking my fork from your readme? If you think micropython may be too niche, I also understand not wanting to clutter the readme.
Thanks!
Please consider the following snippet:
import tomli
import tarfile
with tarfile.open("test.tar") as tar:
f = tar.extractfile("test.toml")
assert f is not None
tomli.load(f)
When I run mypy on it, I get:
test.py:7: error: Argument 1 to "load" has incompatible type "IO[bytes]"; expected "BinaryIO" [arg-type]
Found 1 error in 1 file (checked 1 source file)
To be honest, typing docs don't really say what the difference between IO[bytes]
and BinaryIO
is. Besides type stubs, the code works. FWICS, tomllib would suffer from the same problem if not for the fact that mypy overrides its type stub to:
def load(__fp: SupportsRead[bytes], *, parse_float: Callable[[str], Any] = ...) -> dict[str, Any]: ...
I am migrating from toml. And tomli claims full compatibility. But with toml I was able to:
load('some.toml')
but with tomli I get:
tomli.load('some.toml')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.11/site-packages/tomli/_parser.py", line 59, in load
b = __fp.read()
^^^^^^^^^
AttributeError: 'str' object has no attribute 'read'
Both uiri/toml and samuelcolvin/rtoml support filenames in their load
functions. They are then responsible for opening the file with the correct encoding and reading its contents, before calling loads
.
However, tomli
only supports file-like objects, leaving it to the user to open the file and, inevitably, forgetting to use UTF-8 as the encoding (e.g. https://github.com/pulls?q=is%3Apr+author%3Adomdfcoding+UTF-8).
I propose copying toml
and rtoml
and adding support for strings and os.PathLike
objects. That way the user can write:
config = tomli.load("pyproject.toml")
or
config = tomli.load(pathlib.Path("pyproject.toml"))
without having to worry about the encoding, and saving a line over the with open
approach.
What do you think? If you're happy with the idea I'll submit a PR
Please take a look at the discussion
marzer/tomlplusplus#113
I used to manage dependency by pip-compile and requirements.in .
But I can not install tomli via pip-compile.
$ pip-compile
Could not find a version that matches tomli (from -r requirements.in (line 4))
No versions found
Was https://pypi.org/simple reachable?
$ python3
Python 3.9.6 (default, Aug 15 2021, 21:24:28) `
When parsing the follow file:
a = "b\nc" d = 'e\nf' "g\nh" = 1 'i\nj' = 2
i have the result:
{'a': 'b\nc', 'd': 'e\\nf', 'g\nh': 1, 'i\\nj': 2}
Hello.
We are writing tool called micropipenv (https://github.com/thoth-station/micropipenv) and we are trying to make it compatible with various toml backends.
Would it make sense to create an alias for TOMLDecodeError - TomlDecodeError to make it the same as it is in this implementation: https://github.com/uiri/toml/blob/3f637dba5f68db63d4b30967fedda51c82459471/toml/__init__.py#L15 ?
After that, it'd be possible to import toml/tomli and use the same exception name no matter what implementation is available.
What do you think about it?
Traceback (most recent call last):
File "/Users/XYZ/venv/bin/black", line 8, in <module>
sys.exit(patched_main())
File "/Users/XYZ/venv/lib/python3.10/site-packages/black/__init__.py", line 1130, in patched_main
main()
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 1061, in main
with self.make_context(prog_name, args, **extra) as ctx:
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 923, in make_context
self.parse_args(ctx, args)
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 1379, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 2364, in handle_parse_result
value = self.process_value(ctx, value)
File "/Users/XYZ/venv/lib/python3.10/site-packages/click/core.py", line 2326, in process_value
value = self.callback(ctx, self, value)
File "/Users/XYZ/venv/lib/python3.10/site-packages/black/__init__.py", line 105, in read_pyproject_toml
config = parse_pyproject_toml(value)
File "/Users/XYZ/venv/lib/python3.10/site-packages/black/files.py", line 95, in parse_pyproject_toml
pyproject_toml = tomli.load(f)
File "/Users/XYZ/venv/lib/python3.10/site-packages/tomli/_parser.py", line 55, in load
s = __fp.read().decode()
AttributeError: 'str' object has no attribute 'decode'. Did you mean: 'encode'?
I noticed setup.py
was removed here, however since tomli
is a required dependency for bootstrapping flit
this should be included to make installing flit
easier for distros that need it for setting up a PEP-517 build environment.
I'd like to build a package for slackware64-current, but I need the setup.py file.
I ran across an incorrect error message while using tomli.loads
. See the snippet below
import tomli
>>> tomli_str = b'test="a"'
>>> tomli.loads(tomli_str)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "venv\lib\site-packages\tomli\_parser.py", line 74, in loads
src = __s.replace("\r\n", "\n")
TypeError: a bytes-like object is required, not 'str'
Note the actual text of of the type error. It indicates it wants "a bytes-like object is required, not 'str'
". Except it was passed a bytes like object and consulting the documentation loads
actually wants a str
object. So error message text is wrong and in fact opposite of what the function actually wants.
I generated this error using tomli 2.0.1
import rtoml
>>> rtoml.loads('t = 1987-07-05t17:45:00z')
{'t': datetime.datetime(1987, 7, 5, 17, 45, tzinfo=<builtins.TzClass object at 0x7f1892986310>)}
import tomli
>>> tomli.loads('t = 1987-07-05t17:45:00z')
Traceback (most recent call last):
File "<input>", line 1, in <module>
tomli.loads('t = 1987-07-05t17:45:00z')
File "/path/to/python3.9/site-packages/tomli/_parser.py", line 136, in loads
raise suffixed_err(
tomli._parser.TOMLDecodeError: Expected newline or end of document after a statement (at line 1, c
olumn 15)
References:
As part of bootstrapping base pep517 stack you now have distutils -> setuptools -> tomli + pep517 + build. Since you are apparently creating tarballs with --no-setup-py, this bootstrap chain is now impossible which makes pep517 a misery for distros (see pypa/pyproject-hooks#125). Can you please consider not using this switch?
I was just testing this on a malformed toml file:
[tool.a]
hi = 1
ho = 2
[tool.*]
ho = 2
he = 3
toml
gave a nice error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/toml/decoder.py", line 134, in load
return loads(ffile.read(), _dict, decoder)
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/toml/decoder.py", line 455, in loads
raise TomlDecodeError("Invalid group name '" +
toml.decoder.TomlDecodeError: Invalid group name '*'. Try quoting it. (line 4 column 1 char 23)
But tomli
just gave:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/tomli/_parser.py", line 53, in load
return loads(s, parse_float=parse_float)
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/tomli/_parser.py", line 91, in loads
create_dict_rule(state)
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/tomli/_parser.py", line 229, in create_dict_rule
key = parse_key(state)
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/tomli/_parser.py", line 309, in parse_key
key.append(parse_key_part(state))
File "/Users/henryschreiner/git/software/cibuildwheel/venv/lib/python3.9/site-packages/tomli/_parser.py", line 328, in parse_key_part
raise TOMLDecodeError("Invalid key definition")
tomli._parser.TOMLDecodeError: Invalid key definition
Unlike toml
, there's no location for the error, and no indication of what failed either.
I can not download tomli
because there is a cyclic dependency with flit_core
.
Reproducer:
python3.9 -m venv test
./test/bin/pip download --no-binary=":all:" tomli # fails
./test/bin/pip download --no-binary=":all:" flit_core # fails
Example output:
Collecting tomli
File was already downloaded XXX/tomli-1.2.1.tar.gz
Installing build dependencies ... error
ERROR: Command errored out with exit status 2:
command: XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
cwd: None
Complete output (31 lines):
Collecting flit_core<4,>=3.2.0
Downloading flit_core-3.4.0.tar.gz (27 kB)
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting tomli
Downloading tomli-1.2.1.tar.gz (14 kB)
ERROR: Exception:
Traceback (most recent call last):
File "XXX/test/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 357, in run
resolver.resolve(requirement_set)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
abstract_dist = _get_prepared_distribution(
File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
with req_tracker.track(req):
File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
return next(self.gen)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
self.add(req)
File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
raise LookupError(message)
LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
----------------------------------------
ERROR: Command errored out with exit status 2: XXX/test/bin/python3.9 XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.
Add a use_decimal
flag or some other way to parse TOML floats into Decimal
s instead of float
s. Make sure that no lossy conversion to float
ever happens. Go straight from str
to Decimal
.
As the title says. I have triple checked that I have install tomli, but it still doesn't work.
Traceback:
Traceback (most recent call last):
File "C:\Users\pitch\Desktop\Undefined-bot\general-commands.py", line 2, in <module>
import tomllib
ModuleNotFoundError: No module named 'tomllib'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\pitch\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\discord\ext\commands\bot.py", line 606, in _load_from_module_spec
spec.loader.exec_module(lib)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\pitch\Desktop\Undefined-bot\general-commands.py", line 4, in <module>
import tomli as tomllib
ModuleNotFoundError: No module named 'tomli'
Traceback (most recent call last):
File "C:\Users\pitch\Desktop\Undefined-bot\main.py", line 31, in <module>
bot.load_extension(extension)
File "C:\Users\pitch\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\discord\ext\commands\bot.py", line 678, in load_extension
self._load_from_module_spec(spec, name)
File "C:\Users\pitch\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\discord\ext\commands\bot.py", line 609, in _load_from_ionmodule_spec
raise errors.ExtensionFailed(key, e) from e module_spec
discord.ext.commands.errors.ExtensionFailed: Extension 'general-commands' raised an error: ModuleNotFoundError: No module named 'tomli'
I am also using the default python virtual environment (python -m venv project-venv)
This takes us one step closer to stdlib.
How to generate setup.py file
# Create a file with unicode content
with open(datafile, "w", encoding="GBK") as f:
f.write('"你好" = "世界"\n')
# Use `tomli.load()` to load the file
with open(datafile, "r", encoding="GBK") as f:
loaded = tomli.load(f)
# TypeError: File must be opened in binary mode
with open(datafile, "rb") as f:
loaded = tomli.load(f)
# UnicodeDecodeError('utf-8', b'"\xc4\xe3\xba\xc3" = "\xca\xc0\xbd\xe7"\n', 1, 2, 'invalid continuation byte')
We can only use tomli.loads()
as a workaround.
I wrote out the details at nedbat/coveragepy#1481, as the problem was encountered during use of coveragepy in a Windows CI run. I don't have a local Windows environment handy to test with.
The meat of it is, I have this entry in a TOML file, and I think it's valid, and it parses fine on Linux and MacOS, but not Windows, using tomli 2.0.1:
install = "if [ $VIRTUAL_ENV ]; then pip install -r local-requirements.txt; else printf '%s\n' 'Please activate a venv first'; return 1; fi"
Result on Windows:
Couldn't read config file pyproject.toml: Unescaped '\' in a string (at line 62, column 21)
Yeah .. looks like test suite is not included :/
It seems that I can do this:
with open("./config.toml", "rb") as f:
tomli.load(f)
But not this:
f = Path("./config.toml").read_bytes()
tomli.load(f)
Error message:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/var/folders/jc/rnc4j5c90sx162px6x2gl33h0000gp/T/ipykernel_86569/1174330575.py in <module>
1 f = Path("./config.toml").read_bytes()
----> 2 tomli.load(f)
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tomli/_parser.py in load(__fp, parse_float)
57 def load(__fp: BinaryIO, *, parse_float: ParseFloat = float) -> dict[str, Any]:
58 """Parse TOML from a binary file object."""
---> 59 b = __fp.read()
60 try:
61 s = b.decode()
AttributeError: 'bytes' object has no attribute 'read'
I assume the pathlib.read_bytes
method would be equivalent to with open(path, "rb"
). Any idea why the pathlib method doesn't work with tomli?
The latest recommendations from Python is to use pathlib over the with open
method, so would be nice if this works. Thanks!
Long story short, having a TOML parser in Python stdlib would be great and tomli seems to be the best implementation available right now, so also the best candidate for stdlib. Would you be interested in trying to push it?
The relevant CPython bug is: https://bugs.python.org/issue40059
The doc states:
To build code that uses the standard library if available, but still works seamlessly with Python 3.6+, do the following.
but you bumped requirements to Python 3.7+ in 6a93a19
I wished this could still run on Python 3.6 as advertized which is still supported on many Linux distros.
I noticed tomli
is stricter than toml
when it comes to whitespaces:
$ cat sample.toml
digests = [
{
key = "value",
key2 = "value2"
}
]
$ In [4]: toml.load('sample.toml')
Out[4]: {'digests': [{'key': 'value', 'key2': 'value2'}]}
$ In [5]: tomli.load(open('sample.toml', 'rb'))
...
403 return parse_one_line_basic_str(src, pos)
--> 404 raise suffixed_err(src, pos, "Invalid initial character for a key part")
TOMLDecodeError: Invalid initial character for a key part (at line 2, column 4)
Is it something that can be relaxed, please?
Tomli seems really well put together. I want to make sure it gets maintained. Let me know if there's anything I can do 🙂
Hi! I like the idea of having a simple pure Python library for TOML! And I understand the motivation of keeping it minimal, however, I am not sure if splitting the package into two is the optimal approach. I would like to share my thoughts on that.
dump(s)
, what is the benefit of NOT having dump(s)
? By looking at tomli-w, it seems that the benefit is the absence of ~150 lines of Python code. If it is true, I personally do not see use cases where this benefit can be important. I understand the goal of "theoretically minimal library". Are you sure that this goal is practical?dump(s)
, I have to install two packages instead of one and import two different packages instead of one. This setup does not feel minimal to me (unfortunately, this is exactly my case).loads
can be replaced by using StringIO
and load
:) While this is rather a joke, I would like to say that the "minimalism", when not absolute, becomes a subjective thing.Taking all this into account, merging the two libraries into one looks good to me. What do you think?
This might be an issue with rtoml
, but the benchmark fails with the following error on MacOS:
ModuleNotFoundError: No module named 'setuptools_rust'
Workaround is to remove rtoml
from benchmark/requirements.txt and install it after setuptools-rust
.
$ git diff
diff --git a/benchmark/requirements.txt b/benchmark/requirements.txt
index 2433985..4bff435 100644
--- a/benchmark/requirements.txt
+++ b/benchmark/requirements.txt
@@ -6,4 +6,3 @@ pytomlpp
toml
tomlkit
qtoml
-rtoml
diff --git a/pyproject.toml b/pyproject.toml
index 0aa803f..4c9d2fb 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -96,6 +96,8 @@ description = run the benchmark script against a local Tomli version
deps =
-r benchmark/requirements.txt
commands =
+ pip3 install setuptools-rust
+ pip3 install rtoml
python -c 'import datetime; print(datetime.date.today())'
python --version
python benchmark/run.py
Add this test case once the spec clarifies expected outcome:
[[tab.arr]]
[tab]
arr.val1=1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.