Comments (3)
rtobar
commented on July 30, 2024
@pquentin first of all, thanks for taking the time to open this issue with all the relevant information.
My first instinct to your proposal was to not implement it, because it's not obvious to me that the stack level is constant throughout all the code paths that lead to the warning, and because it's not obvious how to check that that constant value remains correct through time (e.g., if we add an internal indirection layer in the code, etc). However, the level might be a constant value, and adding a unit tests that checks it's value is correct might be possible as well, given some effort. When I get some time (don't hold your breath) I'll do some initial delving I to this and report any findings here. OTOH it might be easier to simply remove the automatic string -> bytes decoding altogether and make a new release.
Out of curiosity, and for better wording in the future: was the warning message not clear enough about the cause of the problem? Other than the file/line associated to the warning (what this issue is about), do you think there would have been something else that could have helped with understanding what was wrong and how to fix it?
from ijson.
pquentin
commented on July 30, 2024
The warning was very clear to me, I just did not know which part of my code raised it. Raising an exception in a new release makes sense. (It's not about me since I no longer have the problem)
from ijson.
rtobar
commented on July 30, 2024
@pquentin Thanks for confirming that the message itself was descriptive, that's good to know.
I finally had some time to have a quick look at this. Sadly, as initially guessed the stack level is not constant throughout all the possible code paths leading to the generation of the warning. This is what I tried out:
diff --git a/ijson/compat.py b/ijson/compat.py
index 4204224..8cf51cd 100644
--- a/ijson/compat.py
+++ b/ijson/compat.py
@@ -45,7 +45,7 @@ mode.
'''
def _warn_and_return(o):
- warnings.warn(_str_vs_bytes_warning, DeprecationWarning)
+ warnings.warn(_str_vs_bytes_warning, DeprecationWarning, stacklevel=6)
return o
def bytes_reader(f):
diff --git a/test/test_base.py b/test/test_base.py
index fcc1eb8..805b75b 100644
--- a/test/test_base.py
+++ b/test/test_base.py
@@ -578,12 +578,18 @@ class IJsonTestsBase(object):
class FileBasedTests(object):
def test_string_stream(self):
- with warning_catcher() as warns:
- events = self.get_all(self.basic_parse, b2s(JSON))
- self.assertEqual(events, JSON_EVENTS)
- if self.warn_on_string_stream:
- self.assertEqual(len(warns), 1)
- self.assertEqual(DeprecationWarning, warns[0].category)
+ for invocation in (
+ (self.basic_parse,),
+ (self.parse,),
+ (self.kvitems, ''),
+ (self.items, ''),
+ ):
+ with warning_catcher() as warns:
+ self.get_all(invocation[0], b2s(JSON), *invocation[1:])
+ if self.warn_on_string_stream:
+ self.assertEqual(len(warns), 1)
+ self.assertEqual(DeprecationWarning, warns[0].category)
+ self.assertIn(__file__, warns[0].filename)
def test_different_buf_sizes(self):
for buf_size in (1, 4, 16, 64, 256, 1024, 4098):The main difference in the tests is that all 4 methods are tested (basic_parse, parse, kvitems and items), and it was expected that the name of the test file appeared in the warning. These are the results (redacted for readibility):
(ijson3.10) [rtobar@bolano ijson]---> pytest -v -k 'test_string_stream'
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0 -- /home/rtobar/venvs/ijson3.10/bin/python3
cachedir: .pytest_cache
rootdir: /home/rtobar/scm/git/ijson
collected 577 items / 565 deselected / 12 selected
test/test_async.py::PythonAsyncTests::test_string_stream FAILED [ 8%]
test/test_async.py::Yajl2AsyncTests::test_string_stream FAILED [ 16%]
test/test_async.py::Yajl2CffiAsyncTests::test_string_stream FAILED [ 25%]
test/test_async.py::Yajl2CAsyncTests::test_string_stream FAILED [ 33%]
test/test_async_types_coroutines.py::PythonAsyncTypesCoroutineTests::test_string_stream FAILED [ 41%]
test/test_async_types_coroutines.py::Yajl2AsyncTypesCoroutineTests::test_string_stream FAILED [ 50%]
test/test_async_types_coroutines.py::Yajl2CffiAsyncTypesCoroutineTests::test_string_stream FAILED [ 58%]
test/test_async_types_coroutines.py::Yajl2CAsyncTypesCoroutineTests::test_string_stream FAILED [ 66%]
test/test_generators.py::PythonGeneratorsTests::test_string_stream PASSED [ 75%]
test/test_generators.py::Yajl2GeneratorsTests::test_string_stream PASSED [ 83%]
test/test_generators.py::Yajl2CffiGeneratorsTests::test_string_stream PASSED [ 91%]
test/test_generators.py::Yajl2CGeneratorsTests::test_string_stream PASSED [100%]
[...]
================================================================================ short test summary info ================================================================================
FAILED test/test_async.py::PythonAsyncTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async.py::Yajl2AsyncTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async.py::Yajl2CffiAsyncTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async.py::Yajl2CAsyncTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async_types_coroutines.py::PythonAsyncTypesCoroutineTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async_types_coroutines.py::Yajl2AsyncTypesCoroutineTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async_types_coroutines.py::Yajl2CffiAsyncTypesCoroutineTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
FAILED test/test_async_types_coroutines.py::Yajl2CAsyncTypesCoroutineTests::test_string_stream - AssertionError: '/home/rtobar/scm/git/ijson/test/test_base.py' not found in '/usr/lib/python3.10/asyncio/base_events.py'
====================================================================== 8 failed, 4 passed, 565 deselected in 0.48s ======================================================================
Similar stack levels (3, 4, 5, 6...) give mixed results too, as well as similar assertions (instead of looking for __file__ for example, I also looked for a more lenient os.path.dirname(os.path.abspath(os.path.normpath(__file__)))).
All in all it looks like there's no easy way out of this, so the warning will remain at its default level, and we'll have to hope that users will be able to spot the faulty invocation on their own. Regardless of this, thanks again for the request and the proposed solution.
from ijson.
Related Issues (20)
- yajl2_c backend crashes on PyPy3 HOT 19
- Is there a way to recursively iterate the key? HOT 4
- ijson.items(file, prefix) waits for EOF HOT 8
- Wheels for Python 3.12 with yajl2_c backend HOT 4
- Include array index HOT 2
- Iterate over more than one prefix? HOT 2
- How to parse a large gzip json file. HOT 2
- Is it possible to use multiple prefix HOT 8
- yajl2_c backend for lambda function HOT 2
- How to use ijson to covert string to dict? HOT 3
- How to read json records in chunks using ijson? HOT 4
- Question: is it possible that returing bytes instead of str could speedup parsing? HOT 3
- Thread safety HOT 9
- Full support for byte stream generator HOT 9
- Allow to use ijson package by a relative import HOT 4
- How can I most-efficiently check for a key in the top-level of a json object? HOT 3
- Python3.12 compilation error: ‘PyGenObject’ has no member named ‘gi_code’ HOT 5
- Is it possible to use isjon with Jsonl, ndjson ? HOT 5
- Memory leak on exception handling with yajl2_c backend HOT 6
- _yajl2 backend broken with Python 3.12 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ijson.