Git Product home page Git Product logo

makibot's Introduction

CI CD

makigas

About

This is the public source code for makigas.es. The whole application is built as a Ruby on Rails monolithic application which allows to browse and watch the videos.

Setting up

Requirements

Requirements:

  • Ruby 3.2 and Bundler. Older Ruby version are not supported.
  • An up to date PostgreSQL database. Other SQL engines are not supported.
  • MeiliSearch, to build the search engine.
  • Node.js + Yarn, for the front-end assets.
  • libpq-dev. If bundle install refuses to install pg, check this.
  • imagemagick. Required for image manipulation on thumbnails and such.
  • A web browser with Selenium support, for running E2E tests.

Getting the code

To install the web application:

$ git clone https://github.com/makigas/makigas.es
$ cd makigas
$ bundle install

Environment variables

Copy the .env.example file to .env.

Tweak the file as you need. For instance, you might need to modify the credentials to access the database if you are in GNU/Linux, because probably your PostgreSQL won't allow unauthenticated connections.

Running the application

Standard rails server, like any other Rails application out there. Because we are using jsbundling-rails, you are expected to be compiling the packs on your own. You can do this running yarn build to build via esbuild once, or yarn build:watch in a separate terminal, to keep doing this in the background and update the packs everytime you save a JS or CSS file.

You can also run bin/guard to start the application via Guard, which will use some guard plugins to keep the application running and up to date, and will recompile the assets every time you change them. Additionally, it will run the test suite and linters whenever you update these files.

Database

Upstream database is PostgreSQL and that is the officially supported one. Said that, this web application may work on MySQL and sqlite3 as well, although I haven't tested this, and it's not officially supported. If you experience bugs by using MySQL, they cannot be fix.

Remember to set the credentials in your .env file if needed for development.

MeiliSearch

We are using MeiliSearch for the full text search index, which can be used to lookup for video content using text that appears in the title, description, transcription and text notes of the episode, with the hope of making it easier to discover and consume.

You will need to have MeiliSearch in your development environment installed if you plan to work in the search system. It is not necessary anymore to keep MeiliSearch always open because indexing has been refered to jobs.

DelayedJob

MeiliSearch indexing has been defered to jobs. If you want to run the jobs, use rake jobs to spawn a DelayedJob server, or use rake jobs:work to run the pending commands as a one-off.

Secrets

makigas.es is using secrets, and it will do until secrets are not supported anymore (let's hope this never happens). While there are newer secret storage systems in Ruby on Rails nowadays, for an open source application I don't understand how that does work, and most big Rails open source projects I've looked for inspiration don't do either.

Production secrets are provided using environment variables, following the Twelve Factor guidelines. In the future I hope to use a different system or to abandon altogether secrets.yml to make this more clear.

Seeds

You can use rails db:seed to initially seed some test data, such as a dashboard user. If you do so, you will be able to log in to the dashboard using [email protected] as username and password as password.

You can fetch live data from makigas.es using the JSON views deployed live, by using makigas:download_production. It may take some time to do so. Not every data structure is downloaded at the moment. Some things might not be included.

Contributing

Read the CONTRIBUTING.md file for more information on how to contribute to the project. Follow the Code of Conduct. My two favourite guidelines from the Contributing file:

  • Send as many issues as you need, but please, keep one topic per issue in order to keep things clean and easy to track.
  • Don't submit surprise PRs with new code. Always discuss your intentions in a tracking issue before starting to work so that we can provide you all the help you need, allocate your idea into the roadmap, or politely reject your idea if we consider it's outside the scope of the project.

If you find a bug in this source code or an issue or visual glitch on the web site, please file a bug. If you find a security vulnerability on this source code, please disclose it in a private way to me. My e-mail address and my PGP key is on my personal website.

License

makigas v6 - source code for the makigas.es application
Copyright (C) 2016-2022 Dani Rodríguez

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Frequently Asked Questions

  • What is the point on sharing the source code? I don't have any particular interest in this source code at this moment. I just want an app that works and that allows me to manage my videos and keep my information up to date.

makibot's People

Contributors

ckmu32 avatar danirod avatar dannywolfmx avatar dependabot-preview[bot] avatar dependabot[bot] avatar izanbf1803 avatar klairm avatar owlnai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

makibot's Issues

Limitar el karma diario

Se sigue haciendo el INSERT como siempre, pero en el SELECT COUNT se cambiará la query SQL por una que limite el puntaje máximo a sumar en el COUNT a 100 por día, reiniciando en la medianoche UTC.

Fix Dockerfile

Dockerfile is currently a mess, basically because I built it back when I did not know much about Dockerfiles. Known issues:

  • It doesn't really make sense to have multiple stage images.
  • It uses ts-node in production instead of transpiling the image.

Set up issue and pull request templates

The goal is to have a way to let users know that:

  • They can use the issue tracker to report bugs.
  • They can use the issue tracker to propose new features.
  • They can open pull requests to fix bugs.
  • They can open pull requests to add features if there has been a discussion yet.

It is important to make clear that opening a PR for features that hasn't been priorly discussed yet and that they come as surprise presents is unpleaseant because someone will have to maintain that code and it may be outside the project roadmap.

Remove !presence command

Bot presence is a debug command that should be removed once it's not needed. The bot should always be available in production. There are better ways to mark the bot as in maintenance mode, which should be explored.

Report current karma

Members would like to know which level they have, or how many points they have.

El antiflood debe ignorar los mensajes enviados al canal de captchas

Recientemente un usuario de Discord, no consciente del funcionamiento del sistema de captchas de nuestro servidor, intentó mandar un mensaje al canal de validaciones antes de validar su cuenta.

Screenshot 2021-02-12 at 12 22 25

Posteriormente a este hecho, mandó un copia y pega del mensaje a un canal real y fue capturado por el sistema antiflood:

Screenshot 2021-02-12 at 12 25 15

Este comportamiento es inconsistente, porque el mensaje original no forma parte de uno de los canales reales del servidor. Mientras un usuario no tenga la cuenta verificada por el sistema de captchas, no está terminado de unir, por lo que ese tipo de mensajes no deben ser tenidos en cuenta para el sistema antiflood.

Solución: el sistema antiflood no debe incluir en su memoria parcial mensajes de miembros no verificados.

Embeber mensajes de Discord

Si se publica un mensaje en un servidor que contiene un link a un mensaje del mismo servidor, el bot debería retirar cualquier posible embed genérico Open Graph que se haya podido presentar a partir del link e incrustar en su defecto un embed que contenga una cita del mensaje publicado.

Migrate settings.db to a better key value storage

Discord.js-Commando provides a settings provider based in a JSON schema that is dumped into a row of an SQLite database.

This is good but I'm cheating a lot by adding a lot of keys, thus it is difficult to scan the JSON document.

This may be better in a custom Setting Provider that stores each item in a database row, maybe allowing for the extra cheats that I use.

An example schema:

  • guild_id: the ID for the tag (for sharding the bot across multiple servers)
  • member_id the ID of the member owning this tag (for keys that depend on each user)
  • key: the name of the tag (for access)
  • value: the value of the tag (string, because then I can store any datatype if I serialize it as JSON)
  • expires_at (null or datetime): time at which the key must expire

Dead during cooldown check

TypeError: Cannot read property 'getTime' of null
    at requiresCooldown (/clank/dist/hooks/verify.js:11:61)
    at VerifyService.handleMessage (/clank/dist/hooks/verify.js:42:13)
    at Makibot.<anonymous> (/clank/dist/hooks/verify.js:22:48)
    at Makibot.emit (events.js:327:22)
    at MessageCreateHandler.handle (/clank/node_modules/discord.js/src/client/websocket/packets/handlers/MessageCreate.js:9:34)
    at WebSocketPacketManager.handle (/clank/node_modules/discord.js/src/client/websocket/packets/WebSocketPacketManager.js:108:65)
    at WebSocketConnection.onPacket (/clank/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:336:35)
    at WebSocketConnection.onMessage (/clank/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:299:17)
    at WebSocket.onMessage (/clank/node_modules/ws/lib/event-target.js:120:16)
    at WebSocket.emit (events.js:315:20)

Automatically add links-disabled role to joining members matching some criteria

Commit 7bdd695 added logic to the antispam system so that it would delete messages sent by members with the links-disabled role containing links.

The next step is to automatically add members to this role if they match some criteria. It is always possible to remove this role later if they are found to be safe. Automatic karma could even do this once a member reaches level 2. Interesting heuristics to use:

  • Members that have in their tags specific words: twitch, YT, YouTube.
  • Members that have non-alphanumeric Unicode characters such as:
    • Emojis: ⚡JohnDoe⚡#1234
    • Manuscript: 𝓳𝓸𝓱𝓷 𝓭𝓸𝓮#1234
    • Bold text: 𝐣𝐨𝐡𝐧 𝐝𝐨𝐞#1234
  • Members that joined Discord less than 24 hours ago.

Log kick events into the private moderation log

If a moderator kicks an user, Clank should send a ModlogEvent into the private modlog channel indicating that the user has been kicked from the server. Additionally, if the mod provided a reason to Discord when issuing the kick, the reason should be logged in the message as well.

The reason is that, seeing why a user was kicked is difficult because it requires opening the audit log in the server settings, plus it is difficult by itself because you might not know that an user was kicked unless you actually reviewed the audit log itself.

Alerts (democratic moderation)

The command !alert allows members with good karma level to report messages. Clank will manage reports and warn or ban users.

Usage spec: !alert <member> [<reason>]. Examples:

  • !alert @danirod: signal an alert for danirod. No specified reason or comment.
  • !alert @danirod spam: signal an alert for danirod, adding spam as a comment reason.

When an alert is received, the following happens:

  • The alert is logged in the private moderation log.
  • Clank updates the alert queue for the member that the alert is targeted to.
  • Clank acknowledges the alert by reacting to the original message instead of replying.

The alert queue is a tag attached to a member. It is a list of timestamps at which an alert has been received.

It is acceptable for multiple people to issue multiple warns when a message should clearly be reported (such as spam). This pushes multiple timestamps into the alert queue of a member. Example:

@BadUser1: join my discord server! https://discord.gg/Mq7TBAB
@VeteranUser1: !alert @BadUser1
@VeteranUser2: !alert @BadUser1 spam
@VeteranUser3: !alert @BadUser1

An automatic warn is issued if a member receives more than N alerts for a 30 minute interval, or more than P alerts for a 4 hour interval. N and P are values that should not be considered hardcoded because they may be tweaked, either algorithmically or manually. Let it be initially N = 3 and P = 7.

An automatic ban is issued if an automatic warn is issued for a member that is already warned.

Interactive captcha journey

Something more friendly, in the style of the KCD bot:

  • Have a private channel between Clank and a member that just joined.
  • Ask some questions, the user replies using the standard yes/no answers or maybe using the Y / N reactions.
  • On successful journies, the member is verified.

Possible script

  1. Hola, $MEMBER$, soy Clank, el bot de moderación de este servidor, y también tu guía para darte la bienvenida y ayudarte a entrar a este servidor. Te voy a hacer una serie de preguntas. Para responder, {{escribe en esta conversación las respuestas sí/no}} | {{reacciona con la S para indicar Sí o con la N para indicar No}}. ¿Entendiste?
  • S => 2
  1. Este servidor busca ser una comunidad abierta y amigable y por ello tenemos un código de conducta de comportamiento. Además, para favorecer un entorno aceptable para todo el mundo tenemos unas normas esenciales. ¿Ya tuviste tiempo de echarle un ojo a las normas?
  • N => 3
  • S => 4
  1. Entonces, por favor, visita #normas-de-uso y asegúrate de estar de acuerdo con todo. Abandona el servidor si no estás de acuerdo. ¿Ya tuviste tiempo de echarle un ojo a las normas?
  • N => 3
  • S => 4
  1. ¡Estupendo! Antes de dejarte entrar, quiero darte algunos consejos. Algunos canales de este servidor están diseñados para que la gente pueda preguntar y responder dudas o pedir ayuda. Si vas a utilizarlos, te hago algunas recomendaciones:
  • No necesitas pedir permiso para preguntar. ¡Para eso estoy aquí! Así que escribe tanto como necesites. Divide tu mensaje en varias partes si hace falta.
  • Lo que tampoco deberías hacer es simplemente enviar mensajes como "¿alguien sabe de Java?". ¡Seguro que si asumes que alguien sabe y planteas tu pregunta directamente resuelves tu problema antes! No se pierde nada por intentarlo.
    ¿Entendiste esto?
  • N => 5
  • S => 6
  1. Consulta https://dontasktoask.com/es/ si quieres saber más. Hicimos el esfuerzo de traducir al castellano el artículo, de hecho. ¡Con esto ya estaría! Cuando {{respondas Sí}} | {{reacciones de nuevo con la S}} podrás ver el resto de canales en la lista. ¡Pásalo bien!
  • S => FIN
  1. ¡Pues ya estamos! Cuando {{respondas Sí}} | {{reacciones de nuevo con la S}} podrás ver el resto de canales en la lista. ¡Pásalo bien!
  • S => FIN

Issues

  • Automatic or manual? (For instance, create a channel as soon as the member joins, or make the member do a trigger such as using the auto-reaction system)
  • How many channels can we keep until we reach the channel limit for temporal members? Should we just close the channel if the member does not interact with the bot for 24 hours to prevent a limit?

Remove !horn command

!horn command will be removed because it is not widely used and it causes a dependency hell with node-opus. !horn command may be reintroduced later on.

Add additional antispam rules

Additional patterns to detect:

  • instagram.com/[a-zA-Z0-9._]+ => detect links to instagram profiles
  • facebook.com/groups/[0-9]+=> detect links to Facebook groups
  • facebook.com/pages/[0-9a-zA-Z._]+/[0-9]+/ => detect links to Facebook pages

The !primo command is broke

Describe the bug
When you make use of the !primo command it can be broked

To Reproduce
Send the message !primo Text`outcuote

Expected behavior
Put the ` charter by first putting a backslash

Screenshots
image

Additional context
Its on the server Message

Log ban events into the private moderation log

If a moderator bans an user, Clank should send a ModlogEvent into the private modlog channel indicating that the user has been banned from the server. Additionally, if the mod provided a reason to Discord when issuing the ban, the reason should be logged in the message as well.

The reason is that, seeing why a user was banned is difficult because it requires opening the audit log in the server settings, plus it is difficult by itself because you might not know that an user was banned unless you actually reviewed the audit log itself.

Split work on modular commands

Currently work is done in Makibot.js. Although this made a great proof of concept, if the bot has to pass the PoC stage, commands should be split in class objects, and there should be some structure so that new commands can easily be added and existing commands can be modified as well.

Useful patterns:

Search on makigas.es

A command may query for videos added to the database.

Tentative command format

!videos [query] (see examples 1, 2)
!video [videoID] (see example 2)

Internal detail

An alternate representation for makigas:video, makigas:playlist and makigas:topic objects was proposed on makigas/makigas.es#75. Although the permalinks are not clear at this moment, it would be possible to get feeds and structured metadata about items on URLs such as https://www.makigas.es/videos.json or https://api.makigas.es/v1/videos.json. This permalink would emit a JSON object the bot can digest. All the heavy load of the query is done by makigas/makigas. This command's scope is limited to sending the proper query to the website and rendering the results.

Concerns

  • Discord will convert links into embeds. Should the bot post the links on the results message and possibly spam multiple links if there are multiple results? Or should the bot provide a numeric ID (see example 2)

Examples

  • Example 1:
danirod: !videos flatmap
Clank: Here are the videos about flatmap I found on makigas.es
  - Scala: flatten y flatMap: www.makigas.es/series/scala/flatten-y-flatmap
  • Example 2:
danirod: !videos flatmap
Clank: Here are the videos about flatmap I found on makigas.es
  - Scala: flatten y flatMap: #183 (use !video 183 for info)
danirod: !video 183
Clank: 183 · flatten y flatMap
  Playlist: Scala. Episode: 18. Length: 8:54
  El flatten permite aplastar los elementos de una lista de listas para crear
  una lista única. Por otra parte, con flatMap podemos combinar un map y un
  flatten: primero aplicamos map a los elementos de una colección de entrada
  y luego le hacemos un flatMap.
  URL: www.makigas.es/series/scala/flatten-y-flatmap
  YouTube ID: youtu.be/JFv9THj85yE

Permitir desactivar el sistema karma

Caso de uso 1: un usuario no quiere participar en el sistema karma, de modo que lo desactiva para que no se cuenten sus mensajes.

Caso de uso 2: moderación desactiva el sistema karma a una cuenta si considera que está utilizando de forma sospechosa el sistema de karma para abusar de sus mecánicas.

En ambos casos,

  • El nivel es siempre 0.
  • La suma de karma siempre da 0.
  • El comando /karma informa de esto.
  • Los canales que requieren nivel superior no dejan escribir si la cuenta no corresponde con un mod.

Antispam exceptions

Add a system to godfather trusted accounts (based on tags or roles). Let the bot look away when inspecting a message if the poster is a trusted account.

Example use case: someone who has been in the server for a month and that is very active may post a link to a Facebook page organically rather than for purposes of doing spam. In such case, having a way to tag (either manually or as part of the karma system) a user as a trusted one so that the message can be posted anyway would be good.

Flood detector

If the system detects the same message being posted in multiple channels by the same user in a very short amount of time, it should mark the message as flooded.

Caveats:

  • Message must match, to avoid having quote replies being tagged as flood.
  • How to handle the buffer? Maybe keeping only the first message and checking the last time the sender has sent a message. Definitely, karma system could help with adding these member properties as tags.

Log wastebin events into the private moderation log

When the 🗑️ emoji is used to delete a priorly wanned message, the bot should send anotice into the private moderation log to mark the warned message as deleted. This should be done to indicate other moderators that the message was deleted, to avoid having them click the message and wonder where it is, or to avoid them spending time thinking if it's worth keeping the warned message or not if another moderator has already chosen so.

Clank should reply to commands using DMs instead of public channels

To avoid being very noisy, Clank should reply the user using DMs instead of publicly mentioning the user in the channel the user typed the message in.

In some cases, such as when triggering the !helper command, Clank may attempt to automatically delete the triggering message to clean up the conversation.

Precog

Have a blocklist based on user IDs, which are assumed to be owned by people with a spammy record, even if they haven’t joined the server yet.

Have the server check the ID of a member that joins the guild against the blocklist. Autoban any member whose ID is contained in the blocklist as soon as they join.

Keep inactive timestamp records for verified GuildMembers

Because verified users have already got a role, Discord's prune tool will not include them when pruning inactive users, which is something that every few months I do for users that haven't logged in, in a long time.

The reason I prune inactive users is that it provides a skewed statistic over the amount of users the server has. Plus, you never know when an inactive account may become a zombie account: it gets compromised and a bad actor uses it to send malware or spam.

The bot should log the last time an user goes available or unavailable in order to remove the verified badge, or at least mark the user as inactive, if the user has not logged in in three months.

If a user gets warned he manualy can delete themselves warn

Describe the bug
Discord if you left a server your roles get reset and if a user lefts it can get back on the server verificate and isnt warned

To Reproduce
As an administrator warn a account you own and left the server and then go back into the server with an invite link
you have veen reseted the roles. If the server has a verification role verify yourself and you are no longer warned

Expected behavior
The bot add the warned role back

Check recent history for the captchas channel looking for users who weren't accepted

Is your feature request related to a problem? Please describe.
After merging #151, the bot is more resilient to errors. Whenever the bot disconnects, the Docker engine will restart the bot very soon in order to make it work again. However, this doesn't mean that users are automatically approved. If the user attempts to type !accept during the downtime of the bot, the user won't be accepted yet. At the moment, this is solved by manually approving the user after the bot gets alive again.

Describe the solution you'd like
The bot should look for recent messages received in the captchas channel and approve those users who verify the following rules:

  1. The user must not have the verify role yet.
  2. The user must still be in the server.
  3. The user must have typed !accept, as usual, or whatever keyword in use.
  4. The user must have been part of the server for at least 5 minutes since the message was posted.

Describe alternatives you've considered
Doing this manually, but it is pain.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.