baptiste0928 / raidprotect Goto Github PK

Actually, the bot uses a custom in-memory cache similar to twilight-cache-inmemory. This allows for high throughput and low latency access to cached data, but the data is not persistent between restarts, which prevents zero-downtime updates.

Storing the cache in Redis would allow restarts without data loss, at the cost of higher latency. I have done some benchmarks using rmp-serde for serialization. It uses slightly more memory than the in-memory cache, but the memory usage is reduced by 50% when using zstd compression (it could be even more by using dictionary compression).

Useful libraries:

• redis: redis client with async support
• bb8: async connection pool
• zstd: zstd compression bindings
• rmp-serde: msgpack implementation w/ serde support

The new version of RaidProtect will focus more on moderation features, with an advanced dashboard to manage sanction of a server. The first step is to implement the basic moderation commands:

• /kick: kick a member from the server
• /ban: ban a user from the server
• /mute: mute a member (using Discord native timeouts)
• /warn: warn a user (send a dm with the warning if the user is on the server)

Implementation

All those commands will have a similar behavior.

1. A moderator run one of the moderation commands.
2. The bot ensures the moderator has required permissions to take action on the targeted user.
3. If no reason has been provided in the command, a modal component prompt the moderator for a reason. If enforce_reason is enabled in guild configuration, the field is required to continue. The moderator can also add internal notes linked to the sanction.
4. The targeted user receives a dm with the reason of the sanction.
5. The bot performs the sanction (kick/ban/mute)
6. A message is sent in the guild logs channel and the sanction is stored in the database.
7. The moderator is informed that the action has been performed with a message.

Because of this similarity, most code can be shared between all the commands.

Progress tracking

• Kick command (steps 1-3 are done)
• Ban command
• Mute command
• Warn command

The new GuildForum channel type is not cached by the bot. Since it's different from other channel types, it should be cached in a new variant of CachedChannel.

Data cached in Redis/KeyDB is compressed using the Zstandard algorithm to reduce memory usage. Since the cached data are small (< 1kb most of the time), compression efficiency could be greatly improved by using compression dictionaries (see The case for Small Data compression in Zstandard documentation).

• Create a script generating random data corresponding to the cached models (with a fixed seed to have a predictable result).
• Use this data to generate a zstd dictionary for each model.
• Using the dictionary in model serialization and deserialization functions (statically linked with the include_bytes! macro).

@baptiste0928 You probably noticed Discord released a few months ago a feature that add a popup where people need to agree the rules before interacting with the server.
Since RaidProtect give a role to the member upon joining, the screen isn't displayed to the member.

The bot should wait for the user to agree rules before sending the captcha.

RaidProtect have actually 4 simple commands. (?help, ?stats, ?link, ?about) It may be interesting to re implement them in a different way :

• /about can replace ?link, ?about and ?help : Users no longer need a /help command, since slash commands are documented. This command could be used to redirect users to the documentation, or to the server support.
  + /stats can be re implemented as it is ( Useless, and complex to implement )

For the command /mute that I hope will happen, it would be necessary to integrate the Discord timeout (instead of being at 60 seconds, 5 minutes, 10 minutes, 1 hour, 1 day or a week, the command can mute a member up to 27 (or 28 I don’t remember) days with timeout functionality. (27 or 28 days is the maximum number of timeouts).
If we want to make mute permanently (without a predefined date) or more than 27/28 days, this will add the role of mute (classic) to a member.

Instead of restricting commands permissions manually, use the new default_member_permissions and dm_permission fields. This will allow users to change command permissions easily and manage them in a unique place.

Blocks on:

• twilight-rs/twilight#1704
• Support in twilight-interactions

Add metrics to the cache, similar to twilight_cache_inmemory::InMemoryCacheStats.

It would be nice to have an anti-phishing feature for dangerous links on Discord.
We can grab the feature from the YAGPDB repository here: https://github.com/botlabs-gg/yagpdb
It is also possible to use the Google safebrowsing integration here: https://safebrowsing.google.com/; more information about this feature here: https://developers.google.com/safe-browsing/v4

Before adding new features to the bot, we need to completely rewrite existing features (at least the most used ones) before August 31th (the deadline is mid-August to allow time for testing and migration of user data).

This issue will be updated regulary to reflect latest progress.

Features:

• Moderation commands (#113)
• Captcha
• Spam detection (#23)
• Channel/server lock
• QoL commands (/help, /profile, ...)

Internal features:

• Caching
• Event and interactions processing
• Database storage
• Internationalization (#65)

Join the development Discord server if you want to help: https://discord.gg/Z3ZWhs38da

Actually, the same service handle both cache, gateway and http proxy. This is bad because each update of one of these three components requires a complete restart of the bot, while these updates are quite frequent due to changes in the API.

The main problem is the cache, which must not stop or data will be lost. However, the cache models do not change often and therefore there will be few restarts related to this component. It is therefore possible to separate the cache in a separate service, to allow restarting the other services without stopping. The cache will also be in charge of storing the ResumeSession to avoid any downtime in case of a gateway restart.

• #25
• Implement cache client
• Store gateway sessions in cache to allow resuming
• #22

We should provide translations for command and command options descriptions. After a survey in our Discord server, we decided to keep command and command options names in English.

twilight_interactions supports localization, but need a function for each key. We can create a simple macro_rules! to generate the functions:

desc_translation!(help_description);

// Expands to:
fn help_description() -> impl IntoIterator<Item = (ToString, ToString)> {
    [
        ("fr", Lang::Fr.help_description()),
        ("en-US", Lang::En.help_description()),
    ]
}

Implement a permission calculator based on the cache data. See twilight_cache_inmemory::permission and twilight_util::permission_calculator.

RaidProtect uses rosetta to handle translation files. Actually, usage of French is hard-coded. The bot should change the display language depending on the user settings.

• For interactions, use the user locale field that Discord sends.
• For other messages, like logs or pm sent to guild members, the locale should be stored in the database along with the guild configuration. The guild_locale field is not reliable.

Implement the guild logs channel, which is used by the bot to inform users of its actions.

• ID stored in the logs_chan configuration field.
• Automatically created if missing, with the raidprotect-logs name (only visible to administrators by default). If another raidprotect-logs channel exists, it will be reused instead.

The anti-spam algorithm performs analysis on messages sent in the last 30 seconds.

Goals

Targets

The algorithm targets the following abusive behaviors:

• Flooding a channel with high amount of messages in a short delay.
• Spam of role and/or user mentions, either by mentioning a lot of users quickly or the same user repeatedly.
• Links, files and/or media spamming, most of the time used to spread malware, self-promote or send abusive content (flashing gifs, ...).

Detection requirements

In order to be as efficient as possible and the least disruptive to members, the algorithm must meet the following requirements:

• Normal conversions between users should not trigger the anti-spam, even when sending multiple messages quickly, like a sentence split across multiple messages.
• When a user has an abusive behavior, the bot must respond quickly (less than 5 messages). This is primary for mentions and link spamming, that can be very annoying for members.

Threat response

In case an abusive behavior is detected, an appropriate response must be given by the bot:

• The user must be sanctioned in order to allow the moderation team to handle the situation. Last messages should also be cleared to remove potential threats such as links.
• The sanction should be proportional to the severity of the abuse. Users that flood should have a small sanction in order to warn them, whereas user that show a deliberate abusive behavior must be muted for a long period, or even banned.
• In case the bot cannot perform a sanction, it should warn moderator team, but does not perform any action such as removing messages, to avoid being rate-limited. It could try to fall back on a lower sanction, such as muting instead of banning.
• Moderators must be informed of the bot actions, using the logs channel. History of deleted message should be kept to allow inspecting the precise detected abuse.

Algorithm implementation

Message analysis

Before processing, some information is extracted from received messages.

• The message content is transformed into ASCII-only string and split in words according to the Unicode specification.
• Links are extracted and categorized (invite, media, other).

Performance

• The algorithm is only run if there are more than 3 messages in the cache.
• Expensive computation results such as message analysis is done only when the message is received, then cached.

Possible improvements

The proposed implementation is a minimal implementation, that does not resist to all circumvention attempts. It is necessary to plan to improve it in the future.

Implement the /kick command. This command is the first important moderation command implemented because it is probably the simplest.

Command usage

• Moderator uses the /kick <user> <optional reason> to kick the user from the guild in which the command is executed.
• The bot checks if the user is a guild member, ensure the moderator has required permissions and the bot can kick the user.
• If no reason has been provided, a modal component prompt the moderator for a reason. If enforce_reason is enabled in guild configuration, the field is required to continue.
• The bot attempts to send a private message to the kicked user with the reason of the kick.
• A message is sent in the guild logs channel and the sanction is stored in guild moderation logs.
• The moderator is informed of the command success with a message.

This version uses new slash commands, but a lot of users wouldn't be aware of this change and thus expect old commands (with ? prefix) to work properly. We should respond to these commands with a message that asks to use the new slash command instead.

Cache performance and memory usage may be improved by relaying on external crates:

• Use smallvec and tinyset to avoid heap allocations
• Consider leapfrog to replace dashmap

Related issues/PR:

• twilight-rs/twilight#1507
• droundy/tinyset#11