Git Product home page Git Product logo

bigfile's Introduction

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文English


Bigfile is a file transfer system, supports http, ftp and rpc protocol. Designed to provide a file management service and give developers more help. At the bottom, bigfile splits the file into small pieces of 1MB, the same slice will only be stored once.

In fact, we built a file organization system based on the database. Here you can find familiar files and folders.

Since the rpc and http protocols are supported, those languages supported by grpc and other languages can be quickly accessed.

More detailed documents can be found here

Features

  • Support HTTP(s) protocol

    • Support rate limit by ip
    • Support cors
    • Support to avoid replay attack
    • Support to validate parameter signature
    • Support Http Single Range Protocol
  • Support FTP(s) protocol

  • Support RPC protocol

  • Support to deploy by docker

  • Provide document with English and Chinese

Install Bigfile

There are kinds of ways to install Bigfile, you can find more detailed documentation here English 简体中文

Start Bigfile

  1. generate certificates

bigfile rpc:make-cert

  1. start server

bigfile multi:server

This will print some information as follows:

    [2019/09/19 15:38:32.817] 56628 DEBUG  bigfile http service listening on: https://0.0.0.0:10985
    [2019/09/19 15:38:32.818] 56628 DEBUG   Go FTP Server listening on 2121
    [2019/09/19 15:38:32.819] 56628 DEBUG  bigfile rpc service listening on: tcp://[::]:10986

HTTP Example (Token Create)

package main

import (
	"fmt"
	"io/ioutil"
	libHttp "net/http"
	"strings"
	"time"

	"github.com/bigfile/bigfile/databases/models"
	"github.com/bigfile/bigfile/http"
)

func main() {
	appUid := "42c4fcc1a620c9e97188f50b6f2ab199"
	appSecret := "f8f2ae1fe4f70b788254dcc991a35558"
	body := http.GetParamsSignBody(map[string]interface{}{
		"appUid":         appUid,
		"nonce":          models.RandomWithMD5(128),
		"path":           "/images/png",
		"expiredAt":      time.Now().AddDate(0, 0, 2).Unix(),
		"secret":         models.RandomWithMD5(44),
		"availableTimes": -1,
		"readOnly":       false,
	}, appSecret)
	request, err := libHttp.NewRequest(
		"POST", "https://127.0.0.1:10985/api/bigfile/token/create", strings.NewReader(body))
	if err != nil {
		fmt.Println(err)
		return
	}
	request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	resp, err := libHttp.DefaultClient.Do(request)
	if err != nil {
		fmt.Println(err)
		return
	}
	if bodyBytes, err := ioutil.ReadAll(resp.Body); err != nil {
		fmt.Println(err)
		return
	} else {
		fmt.Println(string(bodyBytes))
	}
}

RPC Example (Token Create)

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"

	"google.golang.org/grpc"
	"github.com/bigfile/bigfile/rpc"
	"google.golang.org/grpc/credentials"
)

func createConnection() (*grpc.ClientConn, error) {
	var (
		err           error
		conn          *grpc.ClientConn
		cert          tls.Certificate
		certPool      *x509.CertPool
		rootCertBytes []byte
	)
	if cert, err = tls.LoadX509KeyPair("client.pem", "client.key"); err != nil {
		return nil, err
	}

	certPool = x509.NewCertPool()
	if rootCertBytes, err = ioutil.ReadFile("ca.pem"); err != nil {
		return nil, err
	}

	if !certPool.AppendCertsFromPEM(rootCertBytes) {
		return nil, err
	}

	if conn, err = grpc.Dial("192.168.0.103:10986", grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs:      certPool,
	}))); err != nil {
		return nil, err
	}
	return conn, err
}

func main() {
	var (
		err  error
		conn *grpc.ClientConn
	)

	if conn, err = createConnection(); err != nil {
		fmt.Println(err)
		return
	}
	defer conn.Close()
	grpcClient := rpc.NewTokenCreateClient(conn)
    	fmt.Println(grpcClient.TokenCreate(context.TODO(), &rpc.TokenCreateRequest{
    		AppUid:    "42c4fcc1a620c9e97188f50b6f2ab199",
    		AppSecret: "f8f2ae1fe4f70b788254dcc991a35558",
    }))
}

FTP Example

ftp-example

FOSSA Status Get it from the Snap Store

bigfile's People

Contributors

fossabot avatar gamelife1314 avatar zhangjiahao93 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bigfile's Issues

下载源码编译报错

Describe the bug
A clear and concise description of what the bug is.

  • go version: 1.16
  • bigfile version (or commit ref): v1.0.5?
  • operating system: MacOS
  • mysql version: 5.7.x
  • cpu arch: amd64
  • If you use Bigfile in Docker, docker version:
 


## Description

Package GraphicsMagickWand was not found in the pkg-config search path.
Perhaps you should add the directory containing `GraphicsMagickWand.pc'
to the PKG_CONFIG_PATH environment variable
No package 'GraphicsMagickWand' found
pkg-config: exit status 1


## Screenshots

Error 1264: Out of range value for column 'size' at row 1

Describe the bug

FTP 中移动文件时,会爆错

  • go version: 1.12
  • bigfile version (or commit ref): v1.0.7
  • operating system: MacOS?
  • mysql version: 5.7.21
  • cpu arch: amd64
  • If you use Bigfile in Docker, docker version:
 docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:26:49 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:32:21 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Description

移动文件或同时上传多个文件时会报错

Screenshots

image

1.1.x plan

In the next days, we will plan to improve Bigfile. Some new features will come, such as:

  • trash box
  • delete permanently

生成证书出错

Describe the bug
A clear and concise description of what the bug is.

  • go version: no
  • bigfile version (or commit ref): no
  • operating system: no
  • mysql version no
  • docker image version: v1.0.2

Description

运行docker run -v $(pwd):/certs bigfile/bigfile rpc:make-cert --save-to /certs命令时没有生成证书

Screenshots

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.