Per this issue: pobrn/mktorrent#14

I don't exactly understand the issue here, or what the right thing to do might be. Being aware of unicode normalization would be a good idea.

It seems to be a common ask: pobrn/mktorrent#11

I think web and HTTP seeds are the same thing, but it would be good to double check.

Also check imdl torrent create --input .. and other weird combinations.

See: http://bittorrent.org/beps/bep_0052.html

In particular, piece size should be power of two and at least 16KiB. In addition, I should probably bite the bullet and support the v2 spec in its entirety. This will mean implementing BEP0047 padding files.

Save links to relevant references, most likely in the readme for now.

• https://wiki.theory.org/index.php/BitTorrentSpecification
• https://www.bittorrent.org/index.html

I'd like to document alternatives to intermodal, as well as prior art and related applications. Right now this should include other applications and libraries for creating torrents. I should also add notes on when you might want to use something other than intermodal.

Acknowledge that all these apps and libraries were studied for inspiration for intermodal.

Torrent Creating Apps and Libraries

• https://github.com/Rudde/mktorrent
• https://github.com/xxkfqz/pmktorrent
• https://github.com/mukaibot/mktorrent
• https://github.com/rsnitsch/py3createtorrent/
• https://github.com/webtorrent/create-torrent
• https://github.com/RecursiveForest/whatmp3
• https://github.com/torrent-file-editor/torrent-file-editor
• https://github.com/repolho/torrent2magnet
• https://github.com/elektito/ih2torrent
• https://github.com/kz26/dottorrent
• https://github.com/kz26/dottorrent-cli
• https://kimbatt.github.io/torrent-creator/
• https://github.com/rndusr/torf-cli
• https://github.com/webtorrent/create-torrent

Usenet Uploaders

• https://github.com/demanuel/NewsUP/
• https://github.com/animetosho/Nyuu

…to allow overwriting a .torrent file.

• Skip .DS_Store and other common junk files
• Skip files that start with a . or are hidden via attribute
• Skip symlinks unless --follow-symlinks is passed
• Support literal path inclusion and exclusion (paths to specific files, path to dir)
• Support glob inclusion and exclusion
• Support regex inclusion and exclusion

The verbose flag should enable logging, and logging should be expanded to be more useful.

Add bencode subcommands for interacting with bencoded data:

• imdl bencode validate --input PATH validate bencode file(s) at PATH, showing any validation errors encountered
• imdl bencode convert --format yaml PATH convert bencoded files at path to given format.

Should indicate progress for number of files processed, number of bytes (total) processed, and number of bytes (per file) processed.

Draw the file tree incrementally, with progress indicator next to each file.

This should work for torrent creation, and torrent verification.

Started in #213.

Ideally check against other major bencode implementations. It may be good to fuzz it as well.

• Doesn't panic
• Everything that produces a value round-trips to the same buffer
• Check that fails/valids match another implementation

It would be nice to have code coverage information available, generated when CI runs. It should probably run in a separate workflow, to avoid slowing down the build.

The speed of torrent creation and verification could use some work. I picked SHA1 and MD5 implementations basically at random, and there are lots of potential performance bottlenecks in how I hash and verify pieces.

Also, #440 reports performance problems when linking against musl on Linux. If allocation, I/O, or threading is slow with musl, then general performance improvements might cause less allocation and/or I/O (I don't do any threading) which might lessen the performance impact of linking against musl.

First thing will be to set up repeatable benchmarking, so there's stable baseline to compare against. Next there are a bunch of things that are worth trying.

• Split crate into binary and library, to make benchmarking possible
• Set up benchmarking
• Set up flamegraph generation
• Tests which exercise incomplete reads and uneven files, and probably some known good infohashes of large files and directories of files. (They can be generated with an algorithm, so they don't need to be saved in the repo.)
• Test for musl vs libc builds
• Try jemalloc (There might be an issue with musl and jemalloc.)
• Consider unifying hashing and verification. Hashing and verification are implemented separately, but with some work they could be unified. This would be a good idea before pursuing performance improvements that will otherwise need to be duplicated.
• Fix ultra-cringe byte-at-a-time hashing inner loop
• Experiment with buffering more data from each file
• Experiment with reading file contents in a background thread
• Experiment with async I/O
• Experiment with multiple threads
• Make new demo gif with faster speeds
• Consider mmaping files

• imdl torrent create

to do

• test --name, --exclude, --md5sum, --source

• use alloc-less bencode validation
• extract all keys with count

Automatic piece size selection should be based on a goal .torrent file size. Consider making this the default when a piece size is not specified on the command line. (However, when creating a magnet link, it might be better not to specify this.)

I'm not sure if WebTorrent requires or benefits from extra fields, but if so, they should be supported by imdl torrent create.

Currently, squash and merge from the github website removes GPG signatures from commits. Figure out a merge strategy that preserves these signatures. Additionally, write a small script that checks that the history has valid signatures.

• Make sure lints are a hard error in github actions workflow.
• Forbid as conversions
• Consider forbidding println/eprintln, since we should be printing to the environment.
• Lint on missing docs

i.e. 512KiB. For clarity, I think only supporting binary units (kibi mebi, etc) would be best.

BEP is 0019, and metainfo key is url-list. (Actually there is a second BEP, 0017.)

The rules for trailing slashes and single vs. multi-file torrents are somewhat weird, so a nice additional feature would be to issue HEAD requests against the reconstructed URLs to make sure that the files can actually be retrieved.

Here is the relevant part of the spec:

Metadata Extension

In the main area of the metadata file and not part of the "info" section, will be a new key, "url-list". This key will refer to a one or more URLs, and will contain a list of web addresses where torrent data can be retrieved. This key may be safely ignored if the client is not capable of using it.

For example (on multiple lines for readability):
d 8:announce27:http://tracker.com/announce 8:url-list26:http://mirror.com/file.exe 4:info...
If the "url-list" URL ends in a slash, "/" the client must add the "name" from the torrent to make the full URL. This allows .torrent generators to treat this field same for single file and multi-file torrents.

Multi-File Torrents

BitTorrent clients normally use the "name" from the torrent info section to make a folder, then use the "path/file" items from the info section within that folder. For the case of Multi-File torrents, the "url-list" must be a root folder where a client could add the same "name" and "path/file" to create the URL for the request.

For example:
... 8:url-list22:http://mirror.com/pub/ 4:infod5:filesld6:lengthi949e4:pathl10:Readme.txte e4:name7:michael

A client would use all that to build a url: http://mirror.com/pub/michael/Readme.txt

I haven't bothered adding per-flag documentation, but I should rectify this.

Perhaps --allow-uneven-piece-size.

A config file might be useful as a source of defaults for different trackers. It should include a profiles section, which allows setting defaults for different trackers, to be used with imdl torrent create --profile PROFILE

Another useful feature might be configuring file path patterns to ignore.

There should be a single, canonical source of documentation, from which is produced a readme, a man page, and an online book. (Probably blocked on clap-rs/clap#552.)

Per http://www.bittorrent.org/beps/bep_0005.html.

Shill 1: The Shillining.

And actually implement it.

From my own investigations, this includes publisher, publisher-url, and title

When #17 is done, I should use it to find common keys that seem to have a well defined purpose, and add support for them.

I remember seeing a publisher key, for example.

I could either support this with flags per key, or with a JSON/YAML/bencoded blob.

Add continuous integration and release building. Consider Travis, Appveyor, and Github Actions, and any I might have missed.

Test on:

• Linux
• macOS
• Windows

• reuse functionality from #78
• bencode canonical

I'd like to make sure that I don't miss any potentially useful features, since most of the features I've encountered are very easy to implement. I should survey other apps and libraries, including those in #1, as well as the various torrent clients that can be used to create torrents.

Otherwise extra files might be inadvertently included.

I should publish my GPG keys, so that commits and releases can be verified by third parties.

Potential future directions:

• Most likely: A simple and useful single-file format (gzip + tar) plus structured releases with a focus on excellent metadata + web index.html for browser viewing.

• Usenet: Functionality for simultaneously creating usenet and bittorrent releases. Should allow for torrent to be losslessly translated into a NZB, and vice-versa. The ideal flow is: Publisher creates a release with a normal torrent, and also uploads 512KiB blocks of the torrent to usenet. The blocks should be encrypted with a private key distributed with the torrent. Upon receiving the torrent, the whole torrent or missing blocks can be mechanically retrieved from usenet.

• Releases: Functionality for naming and formatting releases. One particularly promising avenue is creating a standard for release names that are both machine readable and human readable. Current release names (like Foo.Bar.S01E17.FMT-GRP) are close to being machine readable, but many different standards exist, so parsing them is awkward and complex. If standardizing those names is politically untenable, then maybe standardizing and adding a machine-readable blob, perhaps bech32 tlv, would be more feasible. Beyond that, an actual schema for the contents of torrents would make building rich services (search, auto-download, library manager, streaming, etc) much easier. For the first type of release, I'm considering albums or research papers.

• S3: Publish a release to an S3 bucket.

• Web: Generate HTML index files so a release can be published to the web.

• Plex/Sonarr/Radarr: Think about how intermodal-generated torrents can be compatible these apps. (Most likely by making sure the metadata is in the right place.)

Print error in red, message in bold. Add always/auto/never --use-color flag.

I.e. -i for --input.

#205 partially addresses this, but I still need to add short flags to imdl torrent create.

In particular, make sure people get a good sense that imdl has expansive future goals, but right now is very usable to create torrents.

Check for filenames that might cause problems on different systems and throw an error, unless a flag is passed. I'm not sure how strict to be. Some ideas for checks, some which are clearly too strict:

• Only allow ascii: Since non-ascii encoding may be buggy
• Don't allow spaces: For shell-safe paths
• Don't allow siblings whose names differ only by case: For case sensitive file systems)
• Only allow /a-zA-Z0-9_.-/: maximum crazy
• Don't allow leading -: conflicts with options and flags
• Dont' allow \/:*?"<>|: I think for windows

In order to learn what features might be useful, I'm interested in finding torrent creation apps currently in use. The Internet Archive creates a lot of torrents, but I'm not sure what they they use.

I'm particularly interested in their use of esoteric and recent BEPs. Jason Scott might be able to help.

Not sure what the flag should be. Maybe --piece-size-power-of-two N or --piece-size-log-two N