Git Product home page Git Product logo

gifreversingbot's Introduction

GifReversingBot

Reddit bot that reverses gifs. Currently running under /u/gifreversingbot.

Setup

Create a file named credentials.ini in the root directory with the following content

[general]
mode = development|production
operator = username to be pinged on crash/other stuff

[database]
type = sqlite|mysql
# Settings for mysql
host = 
username =
password = 
database =

[reddit]
client_id = Reddit client id
client_secret = Reddit client secret
username = Reddit username
password = Reddit password

[imgur]
imgur_id = Imgur api id
imgur_secret = Imgur api secret
imgur_cookie = Cookie generated from an Imgur upload

[gfycat]
gfycat_id = gftcat id
gfycat_secret = gfycat secret

[streamable]
email = streamable account email
password = password

[catbox]
hash = your catbox account hash

From there run python main.py from the root directory to start. GifReversingBot requires Python 3.6+.

You will also need FFmpeg, FFprobe, and gifski binaries on the path or in the same directory.

Commentary

Here's some notes on the more interesting parts of the bot.

Context detection

The bot goes through a couple of steps to determine what the user was asking for. First thing it checks is whether or not the user gave a link in the summon comment themselves. Then it looks through the comment chain to see if there was a gif in the parent comments (it will choose the one last referenced in the chain). Finally, it checks the post for a link.

The bot recognizes "re-reverses" (wherein someone tries to have the bot reverse it's own gif) when it encounters it's own comment while searching through parent comments. When this happens, it searches up the chain a bit farther to find the original link and replies with that.

Reversing gifs/mp4s

GifReversingBot uses two different reversing procedures which it chooses based on a few different circumstances. If it is reversing an mp4 (which is the most common gif type nowadays, go figure), it does the reversal process with FFmpeg. For gifs, it exports each frame with FFmpeg and then reassembles them with gifski. Although gifski produces great gifs, it's very slow and so this process is usually avoided. The bot chooses a reversal method by making an educated guess as to whether the source was originally a gif or an mp4.

Gif Host Library

v3 of GifReversingBot introduces the new Gif Host Library, a new implementation of the code used to describe gifs and hosting websites. It aims to share as much code as possible between the sites, reducing implementation efforts as well as surface area for bugs. Hosts define several properties such as how to parse their links, upload capabilities,and limits. Instead of hard coding where the bot should upload gifs for every host, it is now decided dynamically through the properties of a gif, limits of the hosting sites, and other priorities. It's fairly sturdy even though it's new and abstracts away many annoyances, differences, and difficulties managing the gif host sites.

gifreversingbot's People

Contributors

pmdevita avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

gifreversingbot's Issues

Gfycat upload edgecase

File "/home/pmdevita/bots/vredditshare/core/hosts/gfycat.py", line
194, in upload                                                       
    if ticket["task"] == "encoding": 

Imgur crash

Seems Imgur has been having load problems recently

uploading... Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 155, in process_comment
    original_gif_file.audio)
  File "/home/pmdevita/bots/vredditshare/core/hosts/imgur.py", line 349, in upload
    id = imgurupload(file, gif_type, nsfw=nsfw)
  File "/home/pmdevita/bots/vredditshare/core/hosts/imgur.py", line 160, in imgurupload
    print("received wait ticket:", upload['data']['ticket'])
UnboundLocalError: local variable 'upload' referenced before assignment

Large video files require too much RAM to be processed

The server that currently runs GRB doesn't have enough RAM to process some larger requests. By my own estimates, FFMPEG wants at least 10GB of RAM for large requests. Most requests handle fine but some larger jobs are simply impossible.

There are a couple solutions to this

  • Pass these requests (current solution)
  • Download Buy more RAM
  • From what I understand, FFMPEG renders out frames in memory and then concats. Doing it ourselves would allow us to better control storage usage and split up the process into smaller, manageable chunks
  • Determine if FFMPEG has RAM constraints that we can lower so that it doesn't run out of memory attempting the job.

Submission deleted mid-process I think

Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 26, in process_comment
    context = CommentContext(reddit, comment, ghm)
  File "/home/pmdevita/bots/gifreversingbot/core/context.py", line 20, in __init__
    self.url = self.determine_target_url(reddit, self.comment)
  File "/home/pmdevita/bots/gifreversingbot/core/context.py", line 103, in determine_target_url
    url = self.ghm.extract_gif(reddit_object.body, nsfw=self.nsfw)
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/models/reddit/base.py", line 47, in __getattr__
    .format(self.__class__.__name__, attribute))
AttributeError: 'Submission' object has no attribute 'body'

comment deleted before reply

Decided to upload to <class 'core.hosts.gfycat.GfycatHost'> <core.hosts.GifFile
object at 0x7f568e8c29e8>
Not within params of host <class 'core.hosts.redgifs.RedgifsHost'> <core.hosts.G
ifFile object at 0x7f568e8c29e8>
Decided to upload to <class 'core.hosts.catbox.CatboxHost'> <core.hosts.GifFile
object at 0x7f568e8c29e8>
Replying! https://imgur.com/98gbegY.gifv
DELETED_COMMENT: 'that comment has been deleted' on field 'parent' ['__cause__',
 '__class__', '__context__', '__delattr__', '__dict__', '__dir__', '__doc__', '_
_eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__ini
t__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__'
, '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__s
izeof__', '__str__', '__subclasshook__', '__suppress_context__', '__traceback__'
, '__weakref__', 'args', 'error_type', 'field', 'message', 'with_traceback']
Traceback (most recent call last):
  File "main.py", line 62, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 195, in proce
ss_comment
    reply(context, uploaded_gif)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 58, in reply
    raise e
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 44, in reply
    comment = comment.reply(consts.reply_template.format(message))
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/models/reddit/mix
ins/replyable.py", line 26, in reply
    return self._reddit.post(API_PATH['comment'], data=data)[0]
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/reddit.py", line
483, in post
    return self._objector.objectify(data)
  File "/home/pmdevita/.local/lib/python3.7/site-packages/praw/objector.py", lin
e 149, in objectify
    raise APIException(*errors[0])
praw.exceptions.APIException: DELETED_COMMENT: 'that comment has been deleted' o
n field 'parent'
You have new mail in /var/mail/pmdevita
pmdevita@vps225928:~/bots/gifreversingbot$ nano core/secret.py
pmdevita@vps225928:~/bots/gifreversingbot$ python3 main.py

DNS Resolution Error

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='v.redd.it', port=443): Max retries exceeded with url: /6j23cgfef4l61/DASH_480.mp4?source=fallback (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x
7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 62, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 110, in process_comment
    if not new_original_gif.analyze():
  File "/home/pmdevita/bots/vredditshare/core/hosts/reddit.py", line 43, in analyze
    r = requests.get(url)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='v.redd.it', port=443): Max retries exceeded with url: /6j23cgfef4l61/DASH_480.mp4?source=fallback (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at
 0x7f458b4ba880>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Strange error happening on server. Uncommon and reason for happening is uncertain. Seems to happen sometimes on both vreddit and gfycat requests. Is this being caused by the server's DNS failing or something?

Invalid user error

INVALID_USER: 'That user is invalid' on field 'to' {'items': [RedditErrorItem(error_type='INVALID_USER', message='That user is invalid', field='to')]}
Traceback (most recent call last):
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 50, in reply
    comment = comment.reply(consts.reply_template.format(message))
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/models/reddit/mixins/replyable.py", line 38, in reply
    comments = self._reddit.post(API_PATH["comment"], data=data)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 752, in post
    return self._objectify_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 666, in _objectify_request
    self.request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 848, in request
    return self._core.request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/prawcore/sessions.py", line 324, in request
    return self._request_with_retries(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/prawcore/sessions.py", line 260, in _request_with_retries
    raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.Forbidden: received 403 HTTP response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 63, in <module>
    result = process_comment(reddit, reddit.comment(message.id))
  File "/home/pmdevita/bots/gifreversingbot/core/process.py", line 100, in process_comment
    reply(context, gif)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 71, in reply
    reply_message(comment, url)
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 84, in reply_message
    raise e
  File "/home/pmdevita/bots/gifreversingbot/core/reply.py", line 77, in reply_message
    comment.author.message(consts.reply_ban_subject, consts.reply_ban_template.format(url))
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/models/reddit/mixins/messageable.py", line 60, in message
    self._reddit.post(API_PATH["compose"], data=data)
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 752, in post
    return self._objectify_request(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/reddit.py", line 665, in _objectify_request
    return self._objector.objectify(
  File "/home/pmdevita/.local/lib/python3.8/site-packages/praw/objector.py", line 157, in objectify
    raise RedditAPIException(errors)
praw.exceptions.RedditAPIException: INVALID_USER: 'That user is invalid' on field 'to'

Long (30000+) frame gifs create too many parameters for gifski

The solution is to write a Cython library that interfaces directly to the gifski library. Cython is specifically needed because we need to run a second nogil thread for gifski's IO while we feed it frames in the main thread.
This could also let us skip the FFmpeg reverse filter/file rename step

Have Imgur decide file type through 'type' metadata

After all of my guessing, turns out Imgur does say what the original file type was. Grabbing the pic metadata gives the 'type' property, which appears to be the mimetype for the original file. Should probably double check by uploading some stuff ourselves but that should hopefully increase quality of the reversal.

New Gfycat Error

{'errorMessage': '2020-10-11T12:18:49.100Z d3e223b5-7018-4b24-bdac-2685f94592b3 Task timed out after 15.02 sec
onds'}   

Audio reversals

The problem with audio reversals is that

  • If we decide to let users choose, then we could potentially be managing two of every gif which could be a lot of extra processing and increase code complexity

  • If we perform audio reversals always, some people may not want that (might sound weird/be unwanted, might not expect audio)

So, what are the solutions? Are there additional problems that need to be considered?

Better job queueing system

GRB could use a better queueing system.

Currently, the inbox is treated as the queue and while this works, it can be rather crude and it lacks certain features or the ability to add metadata to a video in the queue.

This is somewhat problematic in situations where a gif hosting website goes down. The bot might need to wait for it to come back up but all it can do is either crash or continually retry over and over to reverse and upload. It's a brute-force way of handling the problem, whereas a real queue could make for a more elegant solution.

For example, if Catbox was to go down and a certain job could only be uploaded to that location, we could save the completed reversed gif and add some metadata to the job in the queue, saying we are waiting for Catbox to return online (we should probably also message the operator). GifHost objects could then implement a method for checking if a site is online and the bot could ping every now and then, waiting for the site to come online so it can complete the task.

The job queue would need to have a representation in the database to avoid state-related problems. Make the ID of each job the same as the ID of the incoming message to prevent duplicates in the rare condition it tries to enter a job twice.

Another Imgur 400

4606696
Decided to upload to <class 'core.hosts.imgur.ImgurHost'> <core.hosts.GifFile object at 0x7f1c3ef78240>
getting imgur album id... 1w5nf46
uploading... received wait ticket: 94310288
waiting for processing...I have no idea what's going on
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

^CExiting...

Imgur - mp4 or gifv links?

When Imgur originally started doing mp4 uploads they used a different file extension, .gifv, to represent these files and to push the idea that this was the successor of the GIF.

In the time since then, .mp4 as an extension has caught on due to wider use, and .gifv has dwindled, still only referenced by Imgur now.

So the question is, should we use mp4 links? I think this is probably a yes, it seems more apps and things on the web would likely prefer .mp4 due to this widespread use. I think it would be good to trial it out for a while and see if we get any user responses back complaining about the change.

Reddit video missing crash

Traceback (most recent call last):
  File "main.py", line 55, in <module>
    result = process_comment(reddit, reddit.comment(message.id), q)
  File "/home/pmdevita/bots/vredditshare/core/process.py", line 120, in process_comment
    if not new_original_gif.analyze():
  File "/home/pmdevita/bots/vredditshare/core/hosts/reddit.py", line 26, in analyze
    url = submission.media['reddit_video']['fallback_url']
TypeError: 'NoneType' object is not subscriptable

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.