Git Product home page Git Product logo

adms's Introduction

adms (aasaam distributed media server)

Development mode

This project is under development, and not production ready.

Why

Scaling and maintain file storage is one the most use case. Using AWS/GCS and etc not accessible by all users and some post process of files like converting will be difficult.

Key features

  • Convert Video/Audio to standard HLS.
  • tusd Upload support.
  • Multi domains/buckets.
  • CDN/Secure/Performance in mind design.
  • POSIX Design. For independent any solution
  • Shardable/Distributed Design.
  • Thumbnail generation.

Configurations

All environment variable must has prefix ASM_.

  • Start with ASM_PUBLIC will expose as public configuration, otherwise consideration as secure/private variable.
Public Environment Variable Default Change? Description
ASM_PUBLIC_SERVER_ID 0 ๐Ÿš€ IMPORTANT Server identifier for each deployment, Must be base36 [a-z0-9]{1,32}
ASM_PUBLIC_APP_INSTANCE 2 ๐Ÿ”ง Number of HTTP application instance using PM2
ASM_PUBLIC_SERVER_MAINTAIN false ๐Ÿšซ Is server on maintain mode for just serve no new upload accepted
ASM_PUBLIC_POST_UPLOADED_SIZE_BYTES 4194304 (4MB) ๐Ÿ”ง Max POST body size for accept upload in Mega Bytes, both tus client and HTTP POST
ASM_PUBLIC_TUSD_PATH /resumable-upload ๐Ÿ”ด Path of tusd server for accept tus client requests.
Secure/Private Environment Variable Default Change? Description
ASM_DOWNLOADER_DNS_SERVERS 4.2.2.4,8.8.8.8,1.1.1.1 ๐Ÿ”ง List of dns server that curl will try to resolve
ASM_DOWNLOADER_LIMIT_RATE_KB 256 ๐Ÿ”ง Limit the download speed that curl used in KiloBytes
ASM_TUSD_PORT 1080 ๐Ÿ”ด Port of tusd container to listen behind nginx
ASM_UPLOAD_PORT 8080 ๐Ÿ”ง Nginx port for upload host
ASM_SERVE_PORT 8880 ๐Ÿ”ง Nginx port for upload host
ASM_MAIN_APP_PORT 3000 ๐Ÿ”ด Node.JS main application HTTP port
ASM_AUTH_APP_PORT 3001 ๐Ÿ”ด Node.JS auth application HTTP port
ASM_REDIS_URI redis://adms-redis ๐Ÿ”ด Redis container host
ASM_NGINX_WORKER_PROCESSES 2 ๐Ÿ”ง Nginx worker_processes
ASM_NGINX_WORKER_RLIMIT_NOFILE 2048 ๐Ÿ”ง Nginx worker_rlimit_nofile
ASM_NGINX_WORKER_CONNECTIONS 1024 ๐Ÿ”ง Nginx worker_connections

Change references

This simple emoji show when to change the variables of adms;

  • ๐Ÿš€: Config this variable during launch time and NEVER change it.
  • ๐Ÿ”ง: Config of this could be change any time for changing application workload/behavior.
  • ๐Ÿšซ: Config of this variable may cause service different behavior.
  • ๐Ÿ”ด: Config of this variable is for special use cases or testing, do not touch it if you're not knowing how that works.

Storage

Path Partition # Type Description
/storage/files A Persist IMPORTANT Files store on this path
/storage/tusd A Temporary tusd use for uploaded chunks
/storage/cache X Temporary application use for generate thumbnails and etc...

Storage path

We have server id 0, bucket localhost with file hash 2x7pLVtJRWkwaq that contain simple jpeg file;

Which Path Description
[BUCKET_PATH] /storage/files/0/localhost Path of each bucket
[FILE_DIR] [BUCKET_PATH]/q/kw/2x7pLVtJRWkwaq Path of file directory
[FILE_ORIGINAL] [FILE_DIR]/o.jpg Path of original uploaded file
[FILE_META] [BUCKET_PATH]/q/kw/2x7pLVtJRWkwaq/.m.json Path of file meta data json

Storage notes

  • For best practices is good to use base62 of using mongodb ObjectID for easy integration. (eg 507f1f77bcf86cd799439011 is 2x7pLVtJRWkwaq)
  • [FILE_DIR] use levels=1:2 hash of file like nginx proxy_cache_path level for gain more performance and backup in mind solutions for rsync.

Buckets

Each deployment need bucket files that standard json file

[
  {
    // identifier of buckets
    "id": "localhost",
    // if active and new upload will be proceed otherwise just serve will work
    "active": true,
    // is private that each serve request need JWT token for access data, otherwise all files is public resource.
    "private": false,
    // list of domains that accepted by this bucket
    "domains": [
      "localhost",
      "127.0.0.1"
    ],
    // list of vod qualities that convert to HLS
    "vod": [
      "360p",
      "720p"
    ],
    // list of aod qualities that convert to HLS
    "aod": [
      "128k"
    ],
    // JWT HS256 secret for sign/verify upload token
    "upload_secret": "upload_secret",
    // JWT HS256 secret for delete or get info of server
    "management_secret": "management_secret",
    // JWT HS256 secret for sign/verify serve token
    "serve_secret": "serve_secret"
  }
]

Bucket notes

  • For thumbnail generation JWT token require for prevent DoS/DDoS of cpu usage, even server is public "private": false.
  • Domains must unique across all buckets.

Deployment example

Consider you have sample example.com and you need store media files. We need scale media server to two servers.

Design approach

Scaling and distribute requirement to calculate many parameters.

  • How much you will need for post processors.
  • How much need space for your media.

Private or protected servers

These server could be private or protected via any mechanism like vpn, wireguard, upstream secure or etc.

Host IP Description
adms0 10.0.1.0 adms node 0
adms1 10.0.1.1 adms node 1

Public servers

Consider we have public ip like 5.x.x.x.

up[N].example.com will accept the upload request and process the media and files. s[N].example.com will serve the content via cache in CDN/Reverse Proxy layer for increase your throughput.

Host IP Upstream Description
up0.example.com 5.5.1.0 adms0:ASM_UPLOAD_PORT adms node 0 for accept uploads
up1.example.com 5.5.1.1 adms1:ASM_UPLOAD_PORT adms node 1 for accept uploads
s0.example.com 5.5.2.0 adms0:ASM_SERVE_PORT adms node 0 for serve media (Scale serving 1x)
s0.example.com 5.5.2.1 adms0:ASM_SERVE_PORT adms node 0 for serve media (Scale serving 2x)
s0.example.com 5.5.2.2 adms0:ASM_SERVE_PORT adms node 0 for serve media (Scale serving 3x)
s1.example.com 5.5.2.0 adms1:ASM_SERVE_PORT adms node 1 for serve media (Scale serving 1x)
s1.example.com 5.5.2.1 adms1:ASM_SERVE_PORT adms node 1 for serve media (Scale serving 2x)
s1.example.com 5.5.2.2 adms1:ASM_SERVE_PORT adms node 1 for serve media (Scale serving 3x)

Now you can upload and process in two separate nodes.

For increase high serving content you can add more s[N].example.com node and cache serving content.

adms's People

Contributors

mhf-ir avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.