Comments (8)
~# python3
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getrecursionlimit()
1000
>>>
from falocalrepo.
from falocalrepo.
Wow, never thought there would be a submission that broke the recursion limit o.o
Could you please run these two lines in a Python prompt?
import sys
sys.getrecursionlimit()
I can solve the issue directly in the FAAPI library, but I cannot remove the recursion limit. There is a sys command to do it, but there is no way to predict whether it will cause issues :(
from falocalrepo.
The fox will take longer than I thought. Because of how Python calculates its stack, the recursion fails after less than 500 comments in a chain, that specific function (_remove_recursion
) cannot be avoided or the dict
method will keep going in circles because comments contain an id link to the parent submission/journal.
Never encountered a comments chain that long before so it never popped up.
For the moment the solution is to edit the library manually to remove the problematic line.
Change line 220 in comment.py to:
yield "comments", [] # [dict(_remove_recursion(c)) for c in self.comments]
It's only a temporary fix, but it will stop the recursion error for now.
from falocalrepo.
I applied that patch but I still get an error.
# falocalrepo download submissions 8601571
1/1 0008601571 [#######][#]
Trace written to /mnt/gd-data/FALocalRepo/falocalrepo/FA.log
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8 <module> /usr/local/bin/falocalrepo
> sys.exit(main())
59 main /usr/local/lib/python3.10/dist-packages/falocalrepo/__main__.py
> exit(app.main(standalone_mode=False) or 0)
1055 main /usr/local/lib/python3.10/dist-packages/click/core.py
> rv = self.invoke(ctx)
1657 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return _process_result(sub_ctx.command.invoke(sub_ctx))
1657 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return _process_result(sub_ctx.command.invoke(sub_ctx))
1404 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return ctx.invoke(self.callback, **ctx.params)
760 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return __callback(*args, **kwargs)
26 new_func /usr/local/lib/python3.10/dist-packages/click/decorators.py
> return f(get_current_context(), *args, **kwargs)
316 download_submissions /usr/local/lib/python3.10/dist-packages/falocalrepo/console/download.py
> downloader.download_submissions(list(submission_id))
783 download_submissions /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> self.download_submission(submission_id,
406 download_submission /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> save_comments(self.db, submissions_table, submission.id, submission.comments,
126 save_comments /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> for comment in filter(lambda c: not c.hidden, flatten_comments(comments)):
161 flatten_comments /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})
161 <listcomp> /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})
.... repeats for awhile ....
161 flatten_comments /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})
161 <listcomp> /usr/local/lib/python3.10/dist-packages/faapi/comment.py
Flattens a list of comments. Replies are not modified.
:param comments: A list of Comment objects (flat or tree-structured)
:return: A flat date-sorted (ascending) list of comments
"""
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})
c: 62102234 THE-DALEK-SUPREME
OP
RecursionError:
maximum recursion depth exceeded
from falocalrepo.
You also have to use the --no-comments
option. The FAAPI patch makes it so the comments tree is not traversed when casting the Submission object to a dict. The --no-comments
options makes FALocalRepo ignore comments. this way neither _remove_recursion
nor flatten_comments
are called and the recursion limit isn't reached.
from falocalrepo.
I fixed the issue FurryCoders/FAAPI@97944f1!
Had to completely rewrite the tree-parser in a procedural style, which I really do not like, but alas that's the only solution :(
I will push a new FAAPI release soon and then a patch for FALocalRepo.
from falocalrepo.
Issue is completely fixed and I have also solved another problem caused by the new OP (Original Poster) tags that FA now adds to comments from the author of the submission/journal.
You can download release 4.4.6 to get the fixes :)
from falocalrepo.
Related Issues (12)
- [solved] [incompatible browser cookies] Missing cookies causing session error HOT 11
- Support backends other than furaffinity.net HOT 19
- [Bug]: Crash while downloading submission HOT 1
- [Bug]: Crash when attempting to download a journal HOT 2
- [Bug]: Cookie values aren't being accepted HOT 7
- [Bug]: Downloader doesn't work HOT 3
- [Bug]: urllib3 removed DEFAULT_CIPHERS HOT 2
- [Bug]: Crash when downloading submission without category HOT 1
- [Bug]: Square Brackets Are Removed From Usernames HOT 3
- [Feature] Download gallery folder by ID HOT 1
- [Feature Request]: Option for alternative download structure HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from falocalrepo.