Git Product home page Git Product logo

distpac's Introduction

distpac

A proof of concept package manager backed by p2p transfers.

Disclaimer: This is just a proof of concept and is not intended for actual usage

Design

distpac is meant to be a simplistic package manager composed of both a server and a client portion. The content distribution is set up to distribute packages over the BitTorrent protocol by using transmission-remote and transmission-daemon. From there the server simply runs a tracker, transmission to distribute all packages, and an HTTP server for distributing the package database (an SQLite file). The client uses the package database to get magnet links for different packages and downloads them with transmission.

Installation

Note: all testing for this was done on Arch Linux. It is possible that there are platform specific differences that can cause issues

Server

dist-server expects opentracker, transmission-remote, and transmission-daemon to all be installed on the server along with a recent Rust toolchain. The server specific software is named-file-server and dist-server so you can build and install from the project directory.

$ cargo install --bin named-file-server
$ cargo install --bin dist-server

The server also needs a config file at $XDG_DATA_HOME/distpac/server.yaml. All this currently has is the announce url for the tracker being used like so:

announce_url: http://tracker.address:6969/announce

Client

dist-client expects just transmission-remote and transmission-daemon to be installed along with a recent Rust toolchain. The client can be built and installed from the project directory

$ cargo install --bin dist-client

The client needs a config file at $XDG_DATA_HOME/distpac/client.yaml. This is for the url of the package database server like so:

server_url: http://package.server

Packages

A package is just a directory that follows a specific structure like so

.
├── assets
│  └── …
├── manifest.yaml
└── scripts
   ├── install.sh
   └── uninstall.sh

The assets directory is intended to have the actual content for the package. The manifest file has some metadata on the package like the name and version number (only SemVer style versions are supported). An example manifest is as follows

name: The-Package-Name
version: 1.2.3

There is also the scripts directory that has an install and uninstall script that is intended to be run for installing and uninstalling the package respectively.

Server Overview

The server functions consist of starting and stopping different components as well as adding new packages. To start the components (seeder, tracker server, and database server) you use the start command like so

$ dist-server start

You can also add packages by passing paths to different packages like so

$ dist-server add /path/to/package1 /path/to/package2

Client Overview

The client just consists of syncing the package database and installing and uninstalling packages. Syncing the database is just done with the sync command

$ dist-client sync

While packages can be installed and uninstalled by their name

$ dist-client install example-package
$ dist-client uninstall example-package

distpac's People

Contributors

cosmichorrordev avatar

Stargazers

avatar avatar

Watchers

avatar avatar

distpac's Issues

Store size of packages in db

Since we're using magnet files to distribute the packages we don't get the information on size until after the torrent file is downloaded from peers. This makes it impossible to judge the total download size up-front so it would be much easier if the size was stored separately in a way that can be distributed.

Share transmission code

Both the client and server need to run the transmission-daemon and transmission-remote code, so it would be good to expose them the same way. Maybe as a dist-daemon with a feature flag for extra functionality with the server.

It would also be good to setup a config file for the daemon to restrict the upload speed to make simulating stuff easier.

Server Gameplan

  • Add in skeleton command line
    • Current idea is something like dist-server ((start | stop) [--tracker | --seeder | --database) | (add PACKAGE_PATH …)
    • So basically just three subcommands where two manage running the two different servers used by the server portion and the third adds in new packages
    • May be worth it to eventually add in some basic status command
  • Test out running opentracker for the tracker server so that we can build off this for the actual implementation
  • Test out adding torrents with transmission-daemon and transmission-cli
  • Add in logic for detecting that all required software is installed (Considered unnecessary)
    • opentracker
    • transmission-cli and transmission-daemon
    • intermodal
  • Add in logic for detecting if something is running (Nixed)
    • named-file-server
    • opentracker
    • transmission-daemon
  • Flesh out subcommand functionality
    • start
      • Need to setup distributing the database through a webserver
        • (Experiments of this have already been done with rocket, but look into tide too)
      • Figure out what needs to be done to run transmission-daemon and opentracker
    • stop - Should be as simple as sending signals to all the processes
    • add
      • Add logic for verifying that a package has the correct structure
      • Add in a manifest for the package with some basic metadata
      • Compress all the files
      • Generate the torrent file
      • Add the compressed package and torrent file to a location known by the seeder and add the torrent to the seeder
      • Add the entry to the database
      • Add the infohash to opentracker's whitelist (Nixed for simplicity)

Client Gameplan

  • Add in skeleton CLI
    • My current plan is something like dist-client (sync | ((add | remove) PACKAGE ...))
  • Add in logic for syncing the database
  • Test out transmission-cli for adding torrents from magnet links
  • Need to figure out how to determine when downloads are done
  • Add in logic for installing the packages
    • Extract the package to a staging area Package is sent uncompressed now
    • Run the provided install script
  • Add in logic for removing installed packages
    • Should be as simple as running the uninstall script

Split package db server into separate program

The package db server just acts to server the package database as a named file from a web server. This will end up being an async program and I don't feel like jumping through the hoops of having the whole codebase be async just to accommodate such a small portion so it will be a standalone program just like transmission-cli and opentracker

Restrict package properties

  • Get package name from directory name
    • Remove name from manifest file
  • Prevent whitespace in package name for simplicity's sake

Different version of the same package will conflict

Since the directory name is the same as the package name there will be issues with different versions. To remedy this we can just make the directory name a concatenation of the package name and version

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.