Git Product home page Git Product logo

nzbfs's Introduction

NzbFS

Notice: I haven't used this code for a while now. The big downside to using a usenet-based filesystem is that there is not much you can do if an article is missing. This was the reason I stopped using it. Hopefully this is useful to someone in the future.

Overview

NzbFS is a read-only userspace network filesystem that reads yenc-encoded files from usenet directly. NZBs that are dropped in the filesystem are processed internally and then deleted. A new directory is created with the same name as the NZB without the .nzb extension. The files given by the NZB are listed inside the new folder. Additionally, NzbFS can read files inside uncompressed rars transparently. With NzbFS you should be able to play video/music, mount ISOs, and generally do anything you could with a local file, except data is streamed as needed.

NzbFS should be considered alpha quality. Things may very possibly go wrong, and there is no guarantee that the database serialization format will not change significantly.

Post Processing

After an NZB has been added to NzbFS, a user-provided post-processing script is called with the path of the newly created NZB directory. This can be used to automatically organize files using other programs. Also, by default, RARs are not "extracted" from the NZB until NzbFS is instructed to do so.

$ nzbfs-extract-rars "<mountpoint>/$1"

You could set this up to happen automatically from the post-processing script.

How It Works

Behind the scenes, all information about remote files are stored in the filesystem at another path, the DB root. NzbFS passes most filesystem calls through to the underlying filesystem, and intercepts calls to NZB-related files. Those files are stored in this filesystem database as just the information required to fetch them from the remote server. (Message-IDs, group names) Extended attributes are required for the underlying filesystem to keep track of things like file size.

Network Layer

NzbFS tries to be smart about reading files from the NNTP server: Some caching in memory is done to ensure we aren't hitting the server over and over again for the same article. Typically, you need multiple threads to max out your connection, which NzbFS should use. Additionally, a very rudimentary/hackish readahead system is used to keep data available for client applications.

When reading files, it tries to return data to the client program as early as possible. For example, if the client program only requested the first 1024 bytes of a file, NzbFS will fetch the first segment. As packets are received, the data is yenc decoded and returned to the client, even though the entire segment may not have finished downloading. This helps applications feel more responsive.

Misc

In case you feel a need for even more layers of indirection, you can do things like:

$ cp ~/Downloads/ubuntu.nzb ~/nzb-mnt/
$ sudo mount -o loop ~/nzb-mnt/ubuntu/ubuntu-12.10-desktop-amd64.iso /mnt
$ ls /mnt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.