avakar / pytoml Goto Github PK
View Code? Open in Web Editor NEWA TOML-0.4.0 parser/writer for Python.
License: Other
A TOML-0.4.0 parser/writer for Python.
License: Other
Hello, would it be possible to add an option for pretty printed/indented output?
So we can have:
[[fruit]]
name = "apple"
[fruit.physical]
color = "red"
shape = "round"
[[fruit.variety]]
name = "red delicious"
[[fruit.variety]]
name = "granny smith"
[[fruit]]
name = "banana"
[[fruit.variety]]
name = "plantain"
instead of:
[[fruit]]
name = "apple"
[fruit.physical]
color = "red"
shape = "round"
[[fruit.variety]]
name = "red delicious"
[[fruit.variety]]
name = "granny smith"
[[fruit]]
name = "banana"
[[fruit.variety]]
name = "plantain"
Hi! Your README states that pytoml is deprecated in favor of to toml package, yet I have less issues using pytoml than toml.
So, first of all thanks and congratulations \o/
Also what about removing the statement? It can mislead people towards a broken implementation, while yours is good.
I am trying to use a custom dictionary class for all parsed dictionaries and I thought that I could do that using the translate callback to load/loads. Turns out, translate is only called for scalars and lists, but not for dictionaries (even though there is an elif kind == 'table':
in process_value).
It would also be useful if there was a way to provide the dict class and have it done automatically (uiri/toml has such an option).
Thanks!
Executing this:
Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytoml
>>> pytoml.loads('a="\t"')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win-amd64\egg\pytoml\parser.py", line 23, in loads
File "build\bdist.win-amd64\egg\pytoml\parser.py", line 344, in _p_toml
File "build\bdist.win-amd64\egg\pytoml\parser.py", line 124, in expect_eof
File "build\bdist.win-amd64\egg\pytoml\parser.py", line 164, in _expect
pytoml.core.TomlError: <string>(1, 1): msg
Apparently a tab is not allowed in a string, yielding a uninformative error message. Would be nice to see this fixed to prevent others from stumbling over the same message.
When _Source._expect()
fails it raises a TomlError with the unhelpful error message msg
:
Line 164 in cb92445
Example to provoke this error:
[a]b
The following value types cause load errors:
Example
>>> import pytoml as toml
>>> toml.loads('date = 1979-05-27')
---------------------------------------------------------------------------
TomlError Traceback (most recent call last)
<ipython-input-32-55780aac767f> in <module>()
1 import pytoml as toml
----> 2 toml.loads('date = 1979-05-27')
3
/usr/local/lib/python3.6/site-packages/pytoml/parser.py in loads(s, filename, translate)
21
22 src = _Source(s, filename=filename)
---> 23 ast = _p_toml(src)
24
25 def error(msg):
/usr/local/lib/python3.6/site-packages/pytoml/parser.py in _p_toml(s)
350 stmts.append(_p_stmt(s))
351 _p_ews(s)
--> 352 s.expect_eof()
353 return stmts
354
/usr/local/lib/python3.6/site-packages/pytoml/parser.py in expect_eof(self)
122
123 def expect_eof(self):
--> 124 return self._expect(self.consume_eof())
125
126 def consume(self, s):
/usr/local/lib/python3.6/site-packages/pytoml/parser.py in _expect(self, r)
162 def _expect(self, r):
163 if not r:
--> 164 raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
165 return r
166
TomlError: <string>(1, 12): msg
a.b.c = 1
# Equivalent to:
[a.b]
c = 1
This is a new feature added to what's going to become TOML 1.0:
toml-lang/toml#505
Of course, TOML 1.0 is not released yet, and pytoml currently declares itself a TOML 0.4.0 parser. So it's not a bug that this doesn't work. But I hope that we can update pytoml once the new spec version is finalised. :-)
I have some code parsing TOML doing:
except toml.TomlDecodeError as e:
msg = "'{}' is not a valid TOML file (Error given is: {})\n"
error = e.args[0]
if error == 'Invalid date or number':
msg += (
"One frequent cause of this is forgetting to put quotes "
"around secret keys. Check the file."
)
match = re.search(r"What\? (\w+) ", error)
duplicate = match and next(iter(match.groups()), None)
if duplicate:
msg += (
"One frequent cause of this is using the same account name "
"twice. Check that you didn't use '{}' several times."
).format(duplicate)
ctx.fail(msg.format(secrets_file, e))
All that is because I get bare TomlDecodeError
exceptions.
When raising an exception, we could attach some data so that catching code can make more accurate decision.
E.G:
TomlDecodeError with Invalid date or number
and What? Foo already exists
should have a field
attribute so that we can inspect what data causes the problem.
Also, it's good to have TomlDecodeError
subclasses for each particular problem so that we can have a more granular control over what we catch.
E.G:
class InvalidDateOrNumberTomlError(TomlDecodeError, ValueError):
pass
class DuplicateKeyTomlError(TomlDecodeError, KeyError):
pass
In the end, my ugly hack code could be replaced by:
except InvalidDateOrNumberTomlError as e:
ctx.fail((
'Duplicate key: "{}" '
'One frequent cause of this is using the same account name twice.'
)).format(e.field))
except DuplicateKeyTomlError as e:
ctx.fail((
'Invalid date or number on field "{}" '
'One frequent cause of this is forgetting to put quotes '
'around secret keys. Check the file.'
)).format(e.field))
Which is shorter, clearer, less error prone, and less likely to break in the future.
It's fair to expect that programmers will want to give feedback to their user if parsing a config file fail, so let's make that easy.
Hi!
I was wondering if there would be any possibilities to include commented out pairs. My problem at the moment is that I load a toml file, that has default settings commented out, but left in file for user to simply activate, when doing it manually via nano. Pytoml loads that file, but drops the commented out lines, and when dumping, all these are actually gone...
Hi,
I wonder: what is: https://github.com/avakar/pytoml/blob/master/pytoml/writer.py#L123-L124 for? I recently started to use your library and I've been wondering (until I've been taking a peek into the source) how come that when using tables there's always an "unnecessary newline" prepended.
Knowing the "TOML"-spec, my initial guess was:
(...) Tables (...) appear in square brackets on a line by themselves (...)[1]
Yet, isn't this covered by all the appended '\n's in all calls to io.StringIO() [specifically assigned to 'fout' in pytoml's context]?
Thanks for your work and best regards,
Patrick
When I run
pytoml.dump({'numoffset': datetime.datetime(1977, 6, 28, 7, 32, tzinfo=datetime.timezone(datetime.timedelta(0, -5*60*60)))}, sys.stdout)
it prints
numoffset = 1977-06-28T07:32:00-5.00.0
which is not a valid TOML document.
According to the TOML specification and RFC 3339 section 5.6, the time offset must be formatted differently. For example numoffset = 1977-06-28T07:32:00-05:00
would be a correct TOML encoding.
Note this issue comes up in testcase valid/datetime
from https://github.com/BurntSushi/toml-test
When the data includes a float such as nan
, pytoml writes this out using repr
. However, this isn't valid TOML, and indeed the parser raises an error when trying to parse it. Would it make more sense to raise an error in this situation?
I'm not sure exactly what it is about this line, but it causes pytoml to fail. If I remove the middle third, it will pass, but I'm not sure if it's the quotes, or the parens or something else.
[target.'cfg(not(target_os = "emscripten"))'.dependencies]
Source: https://github.com/Amanieu/thread_local-rs/blob/master/Cargo.toml
Failure:
>>> pytoml.loads(open('Cargo.toml').read())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/vagrant/client-dropbox-python/dropbox-virtual-env-3dfc741c0e6a-mac-intel-10.12/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pytoml/parser.py", line 23, in loads
ast = _p_toml(src)
File "/Users/vagrant/client-dropbox-python/dropbox-virtual-env-3dfc741c0e6a-mac-intel-10.12/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pytoml/parser.py", line 344, in _p_toml
s.expect_eof()
File "/Users/vagrant/client-dropbox-python/dropbox-virtual-env-3dfc741c0e6a-mac-intel-10.12/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pytoml/parser.py", line 124, in expect_eof
return self._expect(self.consume_eof())
File "/Users/vagrant/client-dropbox-python/dropbox-virtual-env-3dfc741c0e6a-mac-intel-10.12/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pytoml/parser.py", line 164, in _expect
raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: <string>(15, 1): msg
It makes packaging slightly easier for downstream.
According to the toml spec, table names follow the same rules for keys which means they can be quoted. The following cases fail:
['bar']
[foo.'bar']
[foo.'bar'.baz]
I am using python 3.5.
Parsing with pytoml.loads()
This toml:
[table]
key = 'val\nue'
Gives out this error
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/home/simon/venv/3.5/lib/python3.5/site-packages/pytoml/parser.py", line 23, in loads
ast = _p_toml(src)
File "/home/simon/venv/3.5/lib/python3.5/site-packages/pytoml/parser.py", line 344, in _p_toml
s.expect_eof()
File "/home/simon/venv/3.5/lib/python3.5/site-packages/pytoml/parser.py", line 124, in expect_eof
return self._expect(self.consume_eof())
File "/home/simon/venv/3.5/lib/python3.5/site-packages/pytoml/parser.py", line 164, in _expect
raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: <string>(3, 1): msg
That toml:
[table]
key = 'val\ue'
Gives out this error:
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 19-21: truncated \uXXXX escape
This toml:
[table]
key = '''value\n'''
Gives out:
{'table': {'key': 'value\n'}}
But should give out (by v0.4.0 specs):
{'table': {'key': 'value\\n'}}
You probably forgot to tag it when uploading to PyPI, due to releasing two versions in quick succession...
I just encountered an issue when trying to pytoml.load
from a python io.StringIO
(or other stream). PR #22 is a quick fix - NB no extra tests added.
When I run
pytoml.dumps({'a': [ [ {} ] ] })
it raises RuntimeError: {}
Perhaps this occurs because the specified data structure can only be encoded as an array of arrays of inline tables. For example a = [ [ {} ] ]
would be a correct TOML encoding.
The TOML specification states that literal-strings are interpreted as-is. Hence, tab characters in literal strings should be possible. Our TOML files contain partial makefile rules and as such require tab characters.
Multi-line literal strings are surrounded by three single quotes on each side and allow newlines. Like literal strings, there is no escaping whatsoever. A newline immediately following the opening delimiter will be trimmed. All other content between the delimiters is interpreted as-is without modification.
However, the decoding fails with the usual generic error. It works in pytoml version 0.1.2
File ".../pytoml/parser.py", line 10, in load
return loads(fin.read(), translate=translate, object_pairs_hook=object_pairs_hook, filename=getattr(fin, 'name', repr(fin)))
File ".../pytoml/parser.py", line 23, in loads
ast = _p_toml(src, object_pairs_hook=object_pairs_hook)
File ".../pytoml/parser.py", line 352, in _p_toml
s.expect_eof()
File ".../pytoml/parser.py", line 124, in expect_eof
return self._expect(self.consume_eof())
File ".../pytoml/parser.py", line 164, in _expect
raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: somefilename(14, 5): msg
The following code raises a TomlError exception both under Python2 and Python3:
import pytoml
pytoml.loads("""
# This is a full-line comment
maximum_error = 4.85e-06 # 1 arcsec
""")
The error message is the following:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/local/lib/python2.7/dist-packages/pytoml/parser.py", line 23, in loads
ast = _p_toml(src)
File "/usr/local/lib/python2.7/dist-packages/pytoml/parser.py", line 344, in _p_toml
s.expect_eof()
File "/usr/local/lib/python2.7/dist-packages/pytoml/parser.py", line 124, in expect_eof
return self._expect(self.consume_eof())
File "/usr/local/lib/python2.7/dist-packages/pytoml/parser.py", line 164, in _expect
raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: <string>(3, 24): msg
Removing the leading zero solves the problem:
import pytoml
pytoml.loads("""
# This is a full-line comment
maximum_error = 4.85e-6 # 1 arcsec
""")
# Result: {'maximum_error': 4.85e-06}
See below:
>>> toml_content
u'["a.com"]\nusername = "\\n0"\npassword = "0"\n'
>>> print(toml_content)
["a.com"]
username = "\n0"
password = "0"
>>> pytoml.loads(toml_content)
{u'a.com': {u'username': u'0', u'password': u'0'}}
The leading \n
in username
is stripped in the pytoml.loads
function.
Currently this is hard-coded:
Line 47 in cd54e91
Would be nice to pass an option to use the '' format for projects where this is the standard
In the PyPI page, there is no description. Does it even support markdown these days?
When I run
pytoml.dump({u"\u00c0": 1}, sys.stdout)
it prints
À = 1
which is not a valid TOML document. Bare keys must only contain ASCII characters.
A quoted string needs to be used, for example "\u00c0" = 1
.
Similar case:
pytoml.dump({u"\u00c0": { "a": 1}}, sys.stdout)
prints
[À]
a = 1
which is also invalid.
pytoml doesn't seem to like it if the very last line has a comment on (and there is no blank line at the end of the file).
Parses ok:
a=1 #comment
Errors:
a=1 #comment
Example:
>>> import pytoml as toml
>>> toml.loads('[table] #comment\n')
{u'table': {}}
>>> toml.loads('[table] #comment')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/pytoml/parser.py", line 23, in loads
ast = _p_toml(src)
File "/usr/local/lib/python2.7/site-packages/pytoml/parser.py", line 344, in _p_toml
s.expect_eof()
File "/usr/local/lib/python2.7/site-packages/pytoml/parser.py", line 124, in expect_eof
return self._expect(self.consume_eof())
File "/usr/local/lib/python2.7/site-packages/pytoml/parser.py", line 164, in _expect
raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: <string>(1, 9): msg
>>> import pkg_resources; pkg_resources.get_distribution("pytoml").version
'0.1.8'
From the spec:
"Keys may be either bare or quoted. Bare keys may only contain letters, numbers, underscores, and dashes (A-Za-z0-9_-)."
Test case:
5="bruce"
Result:
pytoml.core.TomlError: script.bb(1, 1): unexpected
Reason:
By the time pytoml has gotten to the question of whether a token is an id, it has already checked to see if it is a datetime or a float; or failed back to int if it was neither a datetime or a float.
However, what it should do is instead check to see if there's an equals sign (possibly after some amount of whitespace) after the int, and if there is, mark it as an id.
python 3.4.3, pytoml==0.1.7
>>> import pytoml as toml
>>> import datetime
>>> d={'dt':datetime.datetime.now()}
>>> d
{'dt': datetime.datetime(2016, 2, 25, 11, 18, 17, 536767)}
>>> toml.dumps(d)
'dt = 2016-02-25T11:18:17.536767Z\n'
>>> toml.loads(toml.dumps(d))
{'dt': datetime.datetime(2016, 2, 25, 11, 18, 17, 536767, tzinfo=<pytoml.parser._TimeZone object at 0x7f9aca381978>)}
>>> d == toml.loads(toml.dumps(d))
False
Hi,
my understanding is that toml is equivalent to json.
Lets consider following code:
import pytoml as toml
# this is totally serializable to json but pytoml can't handle it
print toml.dumps({'foo': [{'quux': 'quuuux'}, 'bar']})
following trace occurs:
Traceback (most recent call last):
File "./t", line 5, in <module>
print toml.dumps({'foo': [{'quux': 'quuuux'}, 'bar']})
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 11, in dumps
dump(fout, obj, sort_keys=sort_keys)
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 118, in dump
fout.write('{0} = {1}\n'.format(_escape_id(k), _format_value(v)))
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 86, in _format_value
return _format_list(v)
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 49, in _format_list
return '[{0}]'.format(', '.join(_format_value(obj) for obj in v))
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 49, in <genexpr>
return '[{0}]'.format(', '.join(_format_value(obj) for obj in v))
File "/Users/b.palmowski/Library/Python/2.7/lib/python/site-packages/pytoml/writer.py", line 88, in _format_value
raise RuntimeError(v)
RuntimeError: {'quux': 'quuuux'}
toml.loads('bar=["${BAZ}/quux"]')
produces a mangled:
{u'bar': [u'${BAZ']}
The following example is taken from the current (0.4) TOML specification (https://github.com/toml-lang/toml#user-content-comment):
# This is a full-line comment
key = "value" # This is a comment at the end of a line
Parsing it with pytoml
produces an error near the comment on the second line:
import pytoml
pytoml.loads("""
# This is a full-line comment
key = "value" # This is a comment at the end of a line
""")
The error is:
TomlError: <string>(2, 15): msg
I get an error when I try to parse the following valid TOML document:
a = """b "\n c"""
Result: TomlError: <string>(1, 1): msg
Expected: {'a': 'b "\\n c'}
pytoml looks like it's mirroring the API of modules like json and pickle, with load[s]
and dump[s]
methods. But whereas json and pickle have dump(obj, fd)
, pytoml has dump(fd, obj)
.
Obviously fixing this would be a breaking API change, so maybe it's too late. But it's a minor annoyance.
There are currently two TOML parsers for Python. Can the two be merged?
Hello!
There's a discussion happening on distutils-sig about choosing a new standard configuration file format for Python source trees, to be read by tools like pip
when they want to build a package. There are multiple threads, but maybe this one is the most relevant.
Currently sentiment seems to be shifting towards TOML, and attention is focused on pytoml because in some initial testing we found that pytoml seems to handle unicode correctly on py2 and the other toml parser we could find doesn't.
So... I wanted to reach out to you and see what you thought about this. I'm super excited about the possibility of having something like TOML become a standard part of our package building toolkit. But... this would also mean a sudden huge influx of users and attention on your little project here, which I know can be a very mixed blessing. And there'd be issues that come up, like, pip needs support for python 2.6 and 3.3 -- I don't know if you're actually interested in supporting those going forward? (I actually have a patch for this that I'll submit as a PR in a moment, but there are also ongoing costs to keeping old python versions working and a PR doesn't solve that.) And we'd probably need to fix the error message situation. And so forth.
What do you think?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.