Git Product home page Git Product logo

dat-gateway's Introduction

dat-gateway

Stability NPM Version JavaScript Style Guide Build Status Coverage Status

A configurable in-memory Dat-to-HTTP gateway, so you can visit Dat archives from your browser.

If you want a browser that can visit Dat archives, check out Beaker.

Install

To get the dat-gateway command for running your own gateway, use npm:

npm i -g dat-gateway

If you have npx installed, it's even shorter:

npx dat-gateway

Public Gateways:

Usage

You can run dat-gateway to start a gateway server that listens on port 3000. You can also configure it! You can print usage information with dat-gateway -h:

$ dat-gateway -h
dat-gateway

Options:
  --version       Show version number                                  [boolean]
  --config        Path to JSON config file
  --host, -l      Host or ip for the gateway to listen on.  [default: "0.0.0.0"]
  --port, -p      Port for the gateway to listen on.             [default: 3000]
  --dat-port, -P  Port for Dat to listen on. Defaults to Dat's internal
                  defaults.                                      [default: null]
  --dir, -d       Directory to use as a cache.
                                      [string] [default: "~/.cache/dat-gateway"]
  --max, -m       Maximum number of archives allowed in the cache. [default: 20]
  --period        Number of milliseconds between cleaning the cache of expired
                  archives.                                     [default: 60000]
  --ttl, -t       Number of milliseconds before archives expire.
                                                               [default: 600000]
  --redirect, -r  Whether to use subdomain redirects            [default: false]
  --loopback, -L  What hostname to use when serving locally.
                                                      [default: "dat.localhost"]
  -h, --help      Show help                                            [boolean]

You can visit Dat archives through the gateway using a route like this:

http://localhost:3000/{datKey}/{path...}

For example:

http://localhost:3000/40a7f6b6147ae695bcbcff432f684c7bb5291ea339c28c1755896cdeb80bd2f9/assets/img/beaker-0.7.gif

The gateway will even resolve URLs using Dat-DNS:

http://localhost:3000/garbados.hashbase.io/icons/favicon.ico

The gateway will peer archives until they expire from the cache, at which point it proactively halts them and deletes them from disk.

The gateway also supports replicating a hyperdrive instance using websockets

const websocket = require('websocket-stream')
const hyperdrive = require('hyperdrive')

const key = 'c33bc8d7c32a6e905905efdbf21efea9ff23b00d1c3ee9aea80092eaba6c4957'
const url = `ws://localhost:3000/${key}`

const archive = hyperdrive('./somewhere', key)

archive.once('ready', () => {
  const socket = websocket(url)

  // Replicate through the socket
  socket.pipe(archive.replicate()).pipe(socket)
})

Subdomain redirection

By default dat-gateway will serve all dats from the same origin. This means that dats using absolute URLs (starting with /) will be broken. This also means that all dats will share the same localStorage and indexedDB instances which can cause security issues.

In order to resolve these issues, you can use the --redirect flag in conjunction with the host parameter to have each dat served on a subdomain.

For example, http://{host}:{port}/{datkey}/index.html will be redirected to http://{datkey32}.{host}:{port}/index.html which will serve the file from localhost, but at a different domain, ensuring the browser isolates all the contents from each other.

Please note that due to limitations in how URLs work, the dat key will be converted to it's base32 representation instead of hexadecimal using this library

Serving on localhost

Running a gateway locally for personal use is a great idea, but by default dat-gateway uses dat.localhost as its hostname when serving to the local machine. Firefox does not support '*.localhost' domains and so this behavior breaks the gateway for Firefox users.

To fix this, use the -L, --loopback flag to specify localhost as the loopback hostname, like so:

$ dat-gateway -L localhost

This will cause dat-gateway to use localhost as its domain name, which Firefox supports just fine.

Contributions

All contributions are welcome: bug reports, feature requests, "why doesn't this work" questions, patches for fixes and features, etc. For all of the above, file an issue or submit a pull request.

License

Apache-2.0

dat-gateway's People

Contributors

brechtcs avatar chielorenz avatar chrisekelley avatar da2x avatar douganderson444 avatar garbados avatar icarito avatar jalcine avatar jwerle avatar pfrazee avatar rangermauve avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

dat-gateway's Issues

Infinite redirects

Please revert e60a70b and 674d49d.

They causes infinite redirects to /path/////////// in many situations. E.g. when serving with --redirect, the first part can be index.html, or image.jpeg (index.html////// or image.jpeg///// respectively.)

Infinite back-to-back refresh

Steps to reproduce, on single host:

$ dat share

$ dat-gateway

Browse to http://localhost:3000/dathash/
The page refreshes back to back forever. You can see it in the dev tools (or from the flicker). Same problem via deployed gateway on the public server linked from the README.

It appears there is logic designed to have the client wait refreshing until the lastest version of the dat has been downloaded:
dat-ecosystem-archive/hyperdrive-http#25 (comment)

From what I understand this is implemented by having the client send a request with a progressively incrementing version number (in 'wait=' URL param) and have the gateway side respond only if that version has been received, so when client increments to beyond the last version and sends the request, the gateway should not respond to that request.

But, in my case, the client sends requests and gateway responds immediately, and this continues indefinitely. So either the client is sending wrong version number or the gateway has the wrong version number.

Does this reproduce for others?

Cross-host pooling and caching

I'd love to run a few of these on different machines and it would be super cool to be able to configure them such that they don't redundantly store archives

Issues loading all files in browser

System: Ubuntu 18.04.4 LTS

Running a react example on localhost, bundled with browserify, deployed on dat. Attempting to retrieve that dat, but experience this error:

Uncaught SyntaxError: Unexpected token '<'

Opening up the file in the Application tab, I see the index.html inside the javascript bundle.js.

When I try this experiement with dat://beakerbrowser.com/docs, many of the files / assets do not load properly either.

NGINX and Apache config instructions for dat-gateway

Using dat-gateway right means using it behind a vhost like NGINX or Apache, so they can appropriately delegate arbitrary subdomains to the gateway. The README should include instructions on configuring these services.

Deattach the fork

This project is forked from an upstream project that is abandoned and has said that this fork is the upstream project. Please contact GitHub support and ask them to deattach the fork to make this a standalone project. This will enable code search and other features on GitHub and will stop referring people to the now abandoned upstream project.

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Fix Websockets test in Travis

The should handle websockets for replication test hangs forever on Travis. It shouldn't. It should be discovered why, and fixed.

Known archives should be visible to administrator.

Administrators are de facto responsible for the data that crosses their infrastructure. I would like to see a localhost-only stats server that describes the content it knows about: whether it’s actively being peered, how many peers each archive has, etc., and a way to block archives by key so as to prevent their passage over the gateway.

Content in Dat is not content-addressible, so an intrepid jerk could simply re-peer content via a new archive, leaving the administrator to play whack-a-mole... but it’s a start.

Cannot serve all mime types

The gateway can only serve text/html currently. It fails to server gif,png,zip etc, so these file types cannot be downloaded.

Cache & Resource Limits

Currently, archives gathered by the gateway remain in memory until the gateway stops. The gateway does not have a way to dispose of old archives, or to limit the acceptance of new ones. It should.

  • implement a cache

Restore hyperdrive-http versioning capability

Hyperdrive-http uses the "version" query param to checkout a specific version of the dat archive, but all parameters are stripped away before the request is handled by the hyperdrive server:

req.url = `/${path}`

I've tried to use req.url = `/${path}${urlParts.search || ''}` instead and the selected version of the archive is indeed returned to the browser.

To me the versioning system is a very useful feature, is it possible to restore it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.