furrycoders / falocalrepo Goto Github PK

Pure Python program to download submissions, journals, and user folders from the FurAffinity forum in an easily handled database.

License: European Union Public License 1.2

Python 100.00%

furaffinity database scraper python

falocalrepo's People

Contributors

Stargazers

Watchers

Forkers

featherbutt solipsis-project

falocalrepo's Issues

[Bug]: Crash when attempting to download a journal

Version

4.3.0

What happened?

Program crashes when attempting to download a journal.

OS used: Win10 21H2 + Python 3.10.4; Ubuntu 22.04 LTS + Python 3.10.4

How to reproduce the bug?

falocalrepo download users -u lizardlars -f journals

falocalrepo download journals 10105696

Relevant log output

Traceback (most recent call last):
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\__main__.py", line 45, in main
    exit(app.main(standalone_mode=False) or 0)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\console\download.py", line 179, in download_users
    downloader.download_users(list(users), list(folders))
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 686, in download_users
    self._download_users([(u, folders) for u in users])
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 656, in _download_users
    err = self.download_user_journals(user, stop, stop == 1)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 491, in download_user_journals
    err, [entries_added, entries_modified, entries_errors] = self.download_user_folder(
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 412, in download_user_folder
    result, err = download_catch(downloader_entries, user, page)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 104, in download_catch
    return func(*args, **kwargs), 0
  File "D:\Program Files\Python\lib\site-packages\faapi\base.py", line 281, in journals
    for j in (journals := list(map(JournalPartial, info_parsed["sections"]))):
  File "D:\Program Files\Python\lib\site-packages\faapi\journal.py", line 110, in __init__
    self.parse()
  File "D:\Program Files\Python\lib\site-packages\faapi\journal.py", line 125, in parse
    parsed: dict = parse_journal_section(self.journal_tag)
  File "D:\Program Files\Python\lib\site-packages\faapi\parse.py", line 88, in parse_journal_section
    date: datetime = parse_date((t[0] if isinstance(t := tag_date["title"], list) else t).strip())
  File "D:\Program Files\Python\lib\site-packages\dateutil\parser\_parser.py", line 1368, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\dateutil\parser\_parser.py", line 643, in parse
    raise ParserError("Unknown string format: %s", timestr)
dateutil.parser._parser.ParserError: Unknown string format: 4 months ago

[Feature Request]: Option for alternative download structure

The idea

First of all I would like to just say great job on the program so far, it's easy to use in all the ways that matter and on top of all it actually works.
However I would like to request the option for an alternative local file structure. Right now it's all saved as a mixed bag in recursive folders based on the file ID, this is fine and all if you just want a copy and only of one user but when you want to browse them with an image viewer or have multiple users it becomes a bit of a hassle, if at all possible i would like to request the option to at least have it separated by user(i'm personally a bit partial to the format: "(author)/(upload_date) - (title).(ext)") although ideally if it's not too much to ask i would prefer a user defined structure using the values it is already saving to the database anyway.

With that said keep up the good work I really like the program as it currently is already.

Implementation ideas

have a section in the database where the user can either specify one of a set of predefined folder structures or the option to enter a user defined structure

[Bug]: Cookie values aren't being accepted

Version

Latest

What happened?

I was attempting to put in my cookie values to download a gallery and it just ain't accepting the command.I'm not sure if I put it in wrong or what but it ain't going through

How to reproduce the bug?

attempt to put in cookies as shown in the readme

Relevant log output

No response

[Bug]: Crash when downloading submission without category

Version

4.4.4

What happened?

A very small number of submissions don't contain a category

Two examples are https://www.furaffinity.net/view/17027451/ and https://www.furaffinity.net/view/18605987/

Attempting to download either of these submissions results in the error below. This happens because the parsed info_spans object is assumed to have matched at least four spans, but on these submissions there are only three.

How to reproduce the bug?

falocalrepo download submissions 17027451

Relevant log output

tag_category1: Optional[Tag] = tag_info.select_one("span.category-name")
  tag_category2: Optional[Tag] = tag_info.select_one("span.type-name")
  tag_species: Optional[Tag] = (info_spans := tag_info.select("span"))[2]
> tag_gender: Optional[Tag] = info_spans[3]
  tag_description: Optional[Tag] = sub_page.select_one("div.submission-description")
  tag_folder: Optional[Tag] = sub_page.select_one("a.button[href^='/scraps/'],a.button[href^='/gallery/']")

tag_info: <section class="info text">
<div><strong class="highlight">Species</strong> <span>Unspecified / Any</... [283]tag_category1: None
tag_category2: None
tag_species: <span>962 x 604</span>
info_spans: [<span>Unspecified / Any</span>, <span>Multiple characters</span>, <span>962 x 604</span>]
sub_page: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xht... [44920]IndexError:
list index out of range

[Bug]: Crash while downloading submission

Version

4.3.0

What happened?

Crash while downloading submission https://www.furaffinity.net/view/12415691/.

OS used is Win10 21H2. Python version is 3.10.4.

How to reproduce the bug?

falocalrepo download submissions 12415691

Relevant log output

Traceback (most recent call last):
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\__main__.py", line 45, in main
    exit(app.main(standalone_mode=False) or 0)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\console\download.py", line 301, in download_submissions
    downloader.download_submissions(list(submission_id))
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 747, in download_submissions
    self.download_submission(submission_id,
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 374, in download_submission
    self.db.submissions.save_submission({**format_entry(dict(submission), self.db.submissions.columns),
  File "D:\Program Files\Python\lib\site-packages\falocalrepo_database\database.py", line 322, in save_submission
    submission[SubmissionsColumns.FILEEXT.value.name] = SubmissionsColumns.FILEEXT.value.to_entry([
  File "D:\Program Files\Python\lib\site-packages\falocalrepo_database\database.py", line 323, in <listcomp>
    self.save_submission_file(
  File "D:\Program Files\Python\lib\site-packages\falocalrepo_database\database.py", line 348, in save_submission_file
    folder.joinpath(f"{name}{n if n > 0 else ''}" + f".{ext}" * bool(ext)).write_bytes(file)
  File "D:\Program Files\Python\lib\pathlib.py", line 1141, in write_bytes
    with self.open(mode='wb') as f:
  File "D:\Program Files\Python\lib\pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
OSError: [Errno 22] Invalid argument: 'D:\\Program Files\\FALocalRepo\\FA.files\\00\\12\\41\\56\\91\\submission.doc|'

[solved] [incompatible browser cookies] Missing cookies causing session error

Hello,
I have recently come across this program, but have not been able to run it. I've logged into Furaffinty (using the beta theme and without any ad blockers), exported my cookies into a json file (properly named) and placed it in the same directory as the executable file. When I run the program and try to download files it fails at the third check. After examining the cookie file, it seems to be missing the cookies named "n" and "s". I am unable to find which setting in my FA account or which third party cookie provides these. Any suggestions?

[Bug]: urllib3 removed DEFAULT_CIPHERS

Version

4.4.3

What happened?

urllib3 released a backwards incompatible change that removes the DEFAULT_CIPHERS field.

As a result, attempting to run FALocalRepo after a clean install with the current dependencies results in the error below.

To fix this, we need to pin FALocalRepo to an earlier version of urllib3:

poetry add urllib3<2

How to reproduce the bug?

In a fresh poetry environment:

poetry add falocalrepo
poetry install
falocalrepo --version

Relevant log output

Traceback (most recent call last):
  File "C:\Users\...\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\...\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\Scripts\falocalrepo.exe\__main__.py", line 4, in <module>
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\falocalrepo\__init__.py", line 1, in <module>
    from .console import app
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\falocalrepo\console\__init__.py", line 1, in <module>
    from .app import app
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\falocalrepo\console\app.py", line 7, in <module>
    import faapi
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\faapi\__init__.py", line 2, in <module>
    from .base import FAAPI
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\faapi\base.py", line 10, in <module>
    from .connection import CloudflareScraper
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\faapi\connection.py", line 12, in <module>
    from cfscrape import CloudflareScraper  # type: ignore
  File "C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\cfscrape\__init__.py", line 19, in <module>
    from urllib3.util.ssl_ import create_urllib3_context, DEFAULT_CIPHERS
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\falr-05-2023-V98AjTAw-py3.10\lib\site-packages\urllib3\util\ssl_.py)

[Feature] Download gallery folder by ID

according to the docs i can only download the entire gallery, but i want to download a specific folder in the gallery.

[Bug]: Downloader doesn't work

Version

falocalrepo 4.4.1 falocalrepo-database 5.4.3 falocalrepo-server 3.3.3 faapi 3.9.6

What happened?

On November 26th, FA introduced new site layout. FALocalRepo doesn't support it and throws an error when trying to download or update users.

Tested on Windows 10, Python version 3.11.0

How to reproduce the bug?

falocalrepo download users -u trioza -f gallery -f scraps

Relevant log output

Traceback (most recent call last):
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\__main__.py", line 59, in main
    exit(app.main(standalone_mode=False) or 0)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Program Files\Python\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\console\download.py", line 183, in download_users
    downloader.download_users(list(users), list(folders))
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 715, in download_users
    self._download_users([(u, folders) for u in users])
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 687, in _download_users
    err = self.download_user_submissions(user, Folder[folder.lower()], stop, stop == 1)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 555, in download_user_submissions
    err = self.download_user_folder(
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 431, in download_user_folder
    result, err = download_catch(downloader_entries, user, page)
  File "D:\Program Files\Python\lib\site-packages\falocalrepo\downloader.py", line 108, in download_catch
    return func(*args, **kwargs), 0
  File "D:\Program Files\Python\lib\site-packages\faapi\base.py", line 223, in gallery
    info_parsed: dict[str, Any] = parse_user_submissions(page_parsed)
  File "D:\Program Files\Python\lib\site-packages\faapi\parse.py", line 791, in parse_user_submissions
    user_info: dict[str, str] = parse_user_folder(submissions_page)
  File "D:\Program Files\Python\lib\site-packages\faapi\parse.py", line 778, in parse_user_folder
    assert tag_username is not None, _raise_exception(ParsingError("Missing username tag"))
  File "D:\Program Files\Python\lib\site-packages\faapi\exceptions.py", line 56, in _raise_exception
    raise err
faapi.exceptions.ParsingError: Missing username tag

[Bug]: Square Brackets Are Removed From Usernames

Version

4.4.6

What happened?

attempting to download user "l[i]s" becomes "lis" and does not download the proper user data

How to reproduce the bug?

falocalrepo download users -u l[i]s -f gallery -f scraps -f favorites -f journals -f userpage

Relevant log output

falocalrepo download users -u l[i]s -f gallery -f scraps -f favorites -f journals -f userpage
Downloading: lis/gallery
Downloading: lis/scraps
Downloading: lis/favorites
Downloading: lis/journals
Downloading: lis/userpage
lis

Support backends other than furaffinity.net

The idea

FALocalRepo is an incredible tool for downloading and locally organizing art galleries, but it's currently limited in only being able to scrape FurAffinity (or sites with identical HTML). Other gallery sites exist that have an identical or nearly identical data model, and being able to organize content from those galleries in the same local repo would useful.

I'm willing to take on the necessary work for this and submit it as a pull-request.

Implementation ideas

The biggest obstacle is that the FA-specific code in FAAPI is closely coupled to backend-agnostic code, but this can be fixed in two ways.

Create a separate module for each backend site, which implements a standard interface. Allow FALocalRepo invocations to specify a module to load, with FAAPI as the default. This requires no additional changes to FAAPI, but would necessitate some amount of code duplication in each module.
Attempt to decouple FAAPI into its FA-specific and generic parts. The user can inject a dependency into FAAPI's constructor which overrides backend specific behavior.

Neither option would require bloating FAAPI with code for other backends. (2) seems cleaner but would require more work.

Assuming that we want to allow content from multiple backends to coexist in the same local repo, we would also need a "source" field on submissions/users/journals, both to differentiate reused IDs across backends, and as an additional field in searches.

[Bug]: comment recursion error

Version

falocalrepo 4.4.5

What happened?

When running falocalrepo download users --retry 3 --no-comments USER -f gallery -f scraps -f favorites -f journals -f userpage and it running though the users favorites it runs into a submission that has alot of comments and runs into a error

How to reproduce the bug?

falocalrepo download submissions 8601571

Relevant log output

No response

furrycoders / falocalrepo Goto Github PK

falocalrepo's People

Contributors

Stargazers

Watchers

Forkers

falocalrepo's Issues

Version

What happened?

How to reproduce the bug?

Relevant log output

The idea

Implementation ideas

Version

What happened?

How to reproduce the bug?

Relevant log output

Version

What happened?

How to reproduce the bug?

Relevant log output

Version

What happened?

How to reproduce the bug?

Relevant log output

Version

What happened?

How to reproduce the bug?

Relevant log output

Version

What happened?

How to reproduce the bug?

Relevant log output

Version

What happened?

How to reproduce the bug?

Relevant log output

The idea

Implementation ideas

Version

What happened?

How to reproduce the bug?

Relevant log output

Recommend Projects

Recommend Topics

Recommend Org