sanqui / discard Goto Github PK

Let's say if a log file goes over 100mb I want to start partitioning it

Some options:

• Put a day-by-day index into the meta file
• Use bgzip
• Use igzip with pymzML

Discard currently does not support downloading files. Nonetheless, it is clear that a backup of a server is not complete without fetching the attachments. Still, given the possible scope of archival operations, I'm concerned for time spent downloading and disk space, and wouldn't want to enable them by default.

Now, here are some thoughts on how I want to implement downloading files (including avatars, custom emoji ec.).

First, it's important to note that all Discord files are served from a CDN that does not perform any authorization. I can just get the link to an attachment from any server and post it somewhere else and anybody can download it. This means that we don't actually need a specialized tool like Discard to fetch them. It also means they can benefit from being included in the Wayback Machine.
My first step towards supporting files will be similar to the reader: parse a completed run and output a list of URLs for files to download by another tool (e.g. wget or ArchiveBot).

My main question here to potential users of Discard is whether outputting a list of file URLs is enough for you, or if you would prefer Discard to download files as a part of a run. And if so, if you want just the files, or WARCs.

Your input is appreciated!

https://github.com/DiscordWebsite/Logs

Arguments for:

• Standard for HTTP content
• Existing ecosystem
• Indexing support

Arguments against:

• No standardized Websocket support yet. Most data we're getting is over HTTP, but some critical data is over WS and I expect Discord to use WS more in the future. In particular, realtime will rely on it heavily
• Discord API is already entirely JSON
• WARC is primarily intended for websites, this is not a website
• The content we're accessing is authentication-walled, so headers would have to be scrubbed everywhere

Functional requirements

• Archival of selected Discord servers through command-line interface
• Operation using a bot account or a user account
• Output to JSON files
• Basic summary in output (i.e. number of messages and users recorded)
• Halting on error

Non-functional requirements

• Streaming output (not buffered)
• Optional gzip archival of request .json files
• Resilience to network errors
• Scalability to multiple accounts

Technically, Discord doesn't require a user account to join a Discord server and read history. With a public invite and permissive moderation levels on the server, you can join with just a nick, not even a password needed. While spamming open servers with daily puppet accounts is clearly not advisable, archiving through these "light" accounts might reduce load on verified user accounts (for more restricted servers).