pmdevita / gifreversingbot Goto Github PK

View Code? Open in Web Editor NEW

41.0 3.0 4.0 221 KB

A Reddit bot that reverses gifs

Home Page: https://reddit.com/user/gifreversingbot

Python 100.00%

bot reddit-bot gif reverse

gifreversingbot's Issues

Reddit videos mistakenly marked as deleted

https://www.reddit.com/r/de/comments/ljr4u9/der_liebe_hund_meiner_eltern/gnfchy7/?context=3

When GifReversingBot/vredditshare run on this, the bot decides the video was deleted and skips it. I'm guessing that Reddit changed their video API again. Hopefully shouldn't be too difficult to fix.

Should imgur links be i.imgur.com?

Large video files require too much RAM to be processed

The server that currently runs GRB doesn't have enough RAM to process some larger requests. By my own estimates, FFMPEG wants at least 10GB of RAM for large requests. Most requests handle fine but some larger jobs are simply impossible.

There are a couple solutions to this

Pass these requests (current solution)
~~Download~~ Buy more RAM
From what I understand, FFMPEG renders out frames in memory and then concats. Doing it ourselves would allow us to better control storage usage and split up the process into smaller, manageable chunks
Determine if FFMPEG has RAM constraints that we can lower so that it doesn't run out of memory attempting the job.

Have Imgur decide file type through 'type' metadata

After all of my guessing, turns out Imgur does say what the original file type was. Grabbing the pic metadata gives the 'type' property, which appears to be the mimetype for the original file. Should probably double check by uploading some stuff ourselves but that should hopefully increase quality of the reversal.

Is the Bot dead?

The last comments on Reddit from the bot were June 22, 2023.

audio didn't reverse

https://www.reddit.com/r/Unexpected/comments/czythc/face_rubs/ez4iglu/

Gfycat sometimes redirects to Redgifs for SFW gifs

This is a significant problem due to Redgifs running NSFW ads, not a pleasant surprise. My guess is that some of the bot's older gifs were automatically moved to the platform.

Example here: https://www.reddit.com/r/gifsthatendtoosoon/comments/o2siah/that_looks_like_its_straight_out_of_a_cartoon/h287utc/?context=3

Imgur crash

Seems Imgur has been having load problems recently

uploading... Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 155, in process_comment
    original_gif_file.audio)
  File "/home/pmdevita/bots/vredditshare/core/hosts/imgur.py", line 349, in upload
    id = imgurupload(file, gif_type, nsfw=nsfw)
  File "/home/pmdevita/bots/vredditshare/core/hosts/imgur.py", line 160, in imgurupload
    print("received wait ticket:", upload['data']['ticket'])
UnboundLocalError: local variable 'upload' referenced before assignment

Incorrect FPS used for reversal

I thought I already had an issue for this, oops. This is pretty rare so I need to collect some examples to try to figure out what's going on.

https://www.reddit.com/r/reactiongifs/comments/r0jbjs/mrw_its_that_time_of_year_again_to_head_back_to/hltfawg

Gfycat sometimes doesn't hand back json for the ticket get

at this line

ticket = r.json()

there should be a try and then a catch using r.text to figure out what's actually going on

Reupload feature broken

https://www.reddit.com/r/reverseanimalrescue/comments/qpq2xx/couple_speaks_in_tongues_while_shoving_bears_head/hjy32vd/

comment deleted before reply

Decided to upload to <class 'core.hosts.gfycat.GfycatHost'> <core.hosts.GifFile
object at 0x7f568e8c29e8>
Not within params of host <class 'core.hosts.redgifs.RedgifsHost'> <core.hosts.G
ifFile object at 0x7f568e8c29e8>
Decided to upload to <class 'core.hosts.catbox.CatboxHost'> <core.hosts.GifFile
object at 0x7f568e8c29e8>
Replying! https://imgur.com/98gbegY.gifv
DELETED_COMMENT: 'that comment has been deleted' on field 'parent' ['__cause__',
 '__class__', '__context__', '__delattr__', '__dict__', '__dir__', '__doc__', '_
_eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__ini
t__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__'
, '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__s
izeof__', '__str__', '__subclasshook__', '__suppress_context__', '__traceback__'
, '__weakref__', 'args', 'error_type', 'field', 'message', 'with_traceback']
Traceback (most recent call last):
  File "main.py", line 62, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 195, in proce
ss_comment
    reply(context, uploaded_gif)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 58, in reply
    raise e
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 44, in reply
    comment = comment.reply(consts.reply_template.format(message))
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/models/reddit/mix
ins/replyable.py", line 26, in reply
    return self._reddit.post(API_PATH['comment'], data=data)[0]
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/reddit.py", line
483, in post
    return self._objector.objectify(data)
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/objector.py", lin
e 149, in objectify
    raise APIException(*errors[0])
praw.exceptions.APIException: DELETED_COMMENT: 'that comment has been deleted' o
n field 'parent'
You have new mail in /var/mail/pmdevita
pmdevita@vps225928:~/bots/gifreversingbot$ nano core/secret.py
pmdevita@vps225928:~/bots/gifreversingbot$ python3 main.py

DNS Resolution Error

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='v.redd.it', port=443): Max retries exceeded with url: /6j23cgfef4l61/DASH_480.mp4?source=fallback (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x
7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 62, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 110, in process_comment
    if not new_original_gif.analyze():
  File "/home/pmdevita/bots/vredditshare/core/hosts/reddit.py", line 43, in analyze
    r = requests.get(url)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='v.redd.it', port=443): Max retries exceeded with url: /6j23cgfef4l61/DASH_480.mp4?source=fallback (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at
 0x7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Strange error happening on server. Uncommon and reason for happening is uncertain. Seems to happen sometimes on both vreddit and gfycat requests. Is this being caused by the server's DNS failing or something?

Wrong gif returned

https://www.reddit.com/r/bigfoot/comments/njmha1/keep_an_eye_at_the_right_leg_when_it_reaches_back/gz8k59s/

Misread FPS

https://www.reddit.com/r/KidsAreFuckingStupid/comments/j525th/its_my_birthday_and_you_blew_it/g7pe73x/

MediaInfo seems to have miscalculated the FPS, causing the reconstructed GIF to run at a slow speed

Submission deleted mid-process I think

Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 26, in process_comment
    context = CommentContext(reddit, comment, ghm)
  File "/home/pmdevita/bots/gifreversingbot/core/context.py", line 20, in __init__
    self.url = self.determine_target_url(reddit, self.comment)
  File "/home/pmdevita/bots/gifreversingbot/core/context.py", line 103, in determine_target_url
    url = self.ghm.extract_gif(reddit_object.body, nsfw=self.nsfw)
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/models/reddit/base.py", line 47, in __getattr__
    .format(self.__class__.__name__, attribute))
AttributeError: 'Submission' object has no attribute 'body'

Handle 400/500 on directly linked gifs

Right now, the bot expects LinkGifs to just work and doesn't have any error handling for it. Links that don't work should just be skipped.

Imgur - mp4 or gifv links?

When Imgur originally started doing mp4 uploads they used a different file extension, .gifv, to represent these files and to push the idea that this was the successor of the GIF.

In the time since then, .mp4 as an extension has caught on due to wider use, and .gifv has dwindled, still only referenced by Imgur now.

So the question is, should we use mp4 links? I think this is probably a yes, it seems more apps and things on the web would likely prefer .mp4 due to this widespread use. I think it would be good to trial it out for a while and see if we get any user responses back complaining about the change.

Invalid user error

INVALID_USER: 'That user is invalid' on field 'to' {'items': [RedditErrorItem(error_type='INVALID_USER', message='That user is invalid', field='to')]}
Traceback (most recent call last):
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 50, in reply
    comment = comment.reply(consts.reply_template.format(message))
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/models/reddit/mixins/replyable.py", line 38, in reply
    comments = self._reddit.post(API_PATH["comment"], data=data)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 752, in post
    return self._objectify_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 666, in _objectify_request
    self.request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 848, in request
    return self._core.request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/prawcore/sessions.py", line 324, in request
    return self._request_with_retries(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/prawcore/sessions.py", line 260, in _request_with_retries
    raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.Forbidden: received 403 HTTP response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 63, in <module>
    result = process_comment(reddit, reddit.comment(message.id))
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 100, in process_comment
    reply(context, gif)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 71, in reply
    reply_message(comment, url)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 84, in reply_message
    raise e
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 77, in reply_message
    comment.author.message(consts.reply_ban_subject, consts.reply_ban_template.format(url))
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/models/reddit/mixins/messageable.py", line 60, in message
    self._reddit.post(API_PATH["compose"], data=data)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 752, in post
    return self._objectify_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 665, in _objectify_request
    return self._objector.objectify(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/objector.py", line 157, in objectify
    raise RedditAPIException(errors)
praw.exceptions.RedditAPIException: INVALID_USER: 'That user is invalid' on field 'to'

Long (30000+) frame gifs create too many parameters for gifski

The solution is to write a Cython library that interfaces directly to the gifski library. Cython is specifically needed because we need to run a second nogil thread for gifski's IO while we feed it frames in the main thread.
This could also let us skip the FFmpeg reverse filter/file rename step

Disallow automoderator from doing re-reverses

For preventing infinite comment loops. Could be a useful mechanic on other rogue bots too

Another Imgur 400

4606696
Decided to upload to <class 'core.hosts.imgur.ImgurHost'> <core.hosts.GifFile object at 0x7f1c3ef78240>
getting imgur album id... 1w5nf46
uploading... received wait ticket: 94310288
waiting for processing...I have no idea what's going on
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

^CExiting...

Gfycat upload edgecase

File "/home/pmdevita/bots/vredditshare/core/hosts/gfycat.py", line
194, in upload                                                       
    if ticket["task"] == "encoding":

Reddit video missing crash

Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 120, in process_comment
    if not new_original_gif.analyze():
  File "/home/pmdevita/bots/vredditshare/core/hosts/reddit.py", line 26, in analyze
    url = submission.media['reddit_video']['fallback_url']
TypeError: 'NoneType' object is not subscriptable

Augment NSFW tagging by searching Reddit for the gif url

Checking to see if a link has been posted to Reddit and whether the submission has been flagged NSFW could be a useful strategy

New Gfycat Error

{'errorMessage': '2020-10-11T12:18:49.100Z d3e223b5-7018-4b24-bdac-2685f94592b3 Task timed out after 15.02 sec
onds'}

Better job queueing system

GRB could use a better queueing system.

Currently, the inbox is treated as the queue and while this works, it can be rather crude and it lacks certain features or the ability to add metadata to a video in the queue.

This is somewhat problematic in situations where a gif hosting website goes down. The bot might need to wait for it to come back up but all it can do is either crash or continually retry over and over to reverse and upload. It's a brute-force way of handling the problem, whereas a real queue could make for a more elegant solution.

For example, if Catbox was to go down and a certain job could only be uploaded to that location, we could save the completed reversed gif and add some metadata to the job in the queue, saying we are waiting for Catbox to return online (we should probably also message the operator). GifHost objects could then implement a method for checking if a site is online and the bot could ping every now and then, waiting for the site to come online so it can complete the task.

The job queue would need to have a representation in the database to avoid state-related problems. Make the ID of each job the same as the ID of the incoming message to prevent duplicates in the rare condition it tries to enter a job twice.

If we decide to let users choose, then we could potentially be managing two of every gif which could be a lot of extra processing and increase code complexity
If we perform audio reversals always, some people may not want that (might sound weird/be unwanted, might not expect audio)

So, what are the solutions? Are there additional problems that need to be considered?

Wait for rate limit error amount of time

vredditshare was running into this. It should parse out the time limit and then sleep for that amount.

pmdevita / gifreversingbot Goto Github PK

gifreversingbot's Issues

Recommend Projects

Recommend Topics

Recommend Org