anacrolix / confluence Goto Github PK

I think to allow easier processing on client-side, /info endpoint shall return JSON-ified version of info dictionaries.

Accept header can be used to control the format (JSON or bencoded), or a wholly new endpoint can be conceived.

anacrolix/confluence - Docker Image on Docker Hub does not have any published images. Maybe there's something broken (or disabled b/c failures?).
@anacrolix: Would you be interested in setting up this as it was done in anacrolix/dms#97?

As it's being done for -tcpPeers or -utpPeers, for instance.
I've read the code, from which I assume default is false, but found no Default: example help text for sending a PR. I guess the command line parser lib is introspecting the options struct.

seems to happen when I clear the storage the make a request for the file state but the torrent directory has been deleted. I have only tested this on android, would be to get a similar response you would at the data endpoint - file not found message

runtime error: invalid memory address or nil pointer dereference
goroutine 6816 [running]:
net/http.(*conn).serve.func1(0x44201f5a00)
D:/Go/src/net/http/server.go:1491 +0x98
panic(0x75663216c0, 0x442000c040)
D:/Go/src/runtime/panic.go:458 +0x4c0
github.com/anacrolix/torrent.(*File).State(0x0, 0x0, 0x0, 0x0)
D:/Go/projects/src/github.com/anacrolix/torrent/file.go:57 +0x30
github.com/anacrolix/confluence/confluence.fileStateHandler(0x75665868e0, 0x44202a95f0, 0x44200e02d0)
D:/Go/projects/src/github.com/anacrolix/confluence/confluence/handlers.go:80 +0x11c
net/http.HandlerFunc.ServeHTTP(0x75663a4920, 0x75665868e0, 0x44202a95f0, 0x44200e02d0)
D:/Go/src/net/http/server.go:1726 +0x38
github.com/anacrolix/confluence/confluence.withTorrentContext.func1(0x75665868e0, 0x44202a95f0, 0x44200e01e0)
D:/Go/projects/src/github.com/anacrolix/confluence/confluence/middlewares.go:45 +0x498
net/http.HandlerFunc.ServeHTTP(0x4420180cc0, 0x75665868e0, 0x44202a95f0, 0x44200e01e0)
D:/Go/src/net/http/server.go:1726 +0x38
net/http.(*ServeMux).ServeHTTP(0x442017ac90, 0x75665868e0, 0x44202a95f0, 0x44200e01e0)
D:/Go/src/net/http/server.go:2022 +0x178
github.com/anacrolix/confluence/confluence.(*Handler).ServeHTTP(0x442016f230, 0x75665868e0, 0x44202a95f0, 0x44200e01e0)
D:/Go/projects/src/github.com/anacrolix/confluence/confluence/handler.go:19 +0x1a8
net/http.serverHandler.ServeHTTP(0x4420176900, 0x75665868e0, 0x44202a95f0, 0x442045e000)
D:/Go/src/net/http/server.go:2202 +0x154
net/http.(*conn).serve(0x44201f5a00, 0x7566587360, 0x4420318ac0)
D:/Go/src/net/http/server.go:1579 +0xda8
created by net/http.(*Server).Serve
D:/Go/src/net/http/server.go:2293 +0x470

I've noticed that with the 80 half open limit on FreeBSD, the time to gain peers is slow. It seems like confluence is mostly stuck on waiting for connection establishment timeouts if a lot of peers don't have open ports.

Probably MPL2 as well, just like anacrolix/torrent?

Can you please tell me which product and os i choose at google cloud to install this one.
and steps to install.

According to the interface, the path parameter is needed to get the actual data.
Is there any way to list the files in a infohash? How to parse the result of /info?ih= ?

GET /data?ih=&path=.

......the second that it can use WebTorrent.

Any ideas on how to approach an implementation? I know only this: @feross said to "wrap bittorrent in webrtc." Once I have an idea of how to do that, I'll certainly go about trying to do it.

Instead of

GET /data?ih=<infohash in hex>&path=<display path of file declared in torrent info>

what about

GET /data/<infohash in hex>/<display path of file declared in torrent info>

This syntax would also allow us to display "directory listings", when the path following the infohash is not "complete" (i.e. points to a file).

Imagine the following torrent:

/
    dir1/
        a.txt
        b.txt
    dir2/
        c.txt

A GET request to /data/<infohash>/dir1 shall display a directory listing of dir1 with files a.txt and b.txt; just as a regular web server serving static files would do.

Currently, Confluence puts all the cache, Incomplete files and other files where we are executing the program.

It would be nice to be able to specify a data dir where Confluence will store all of its files.

I'm writing a program that's going to download small chunks of lots of different torrents. In many cases, it will time out and then I don't need Confluence to spend any more time trying to find hosts with the file. Could you add a way to stop downloads for a given torrent file so that it doesn't appear in the /status endpoint and its cache is wiped?

like 'http://localhost:8080/info?torrent=https://abc.com/Sintel.torrent'

I limited the requests to 10 concurrent requests to the DHT server and dropped the torrent from the client after collecting the metainfo:

var limiter = make(chan struct{}, 10)
func findTorrent(w http.ResponseWriter, r *http.Request) *torrent.Torrent {
	limiter <- struct{}{}
	t := torrentForRequest(r)
	defer func() {
		<-limiter
		t.Drop()
	}()
	if nowait, err := strconv.ParseBool(r.URL.Query().Get("nowait")); err == nil && nowait {
		select {
		case <-t.GotInfo():
		default:
			http.Error(w, "info not ready", http.StatusAccepted)
			return nil
		}
	} else {
		// w.WriteHeader(http.StatusProcessing)
		select {
		case <-t.GotInfo():
		case <-r.Context().Done():
			return nil
		}
	}

	// w.WriteHeader(http.StatusOK)

	return t
}

Yet the memory usage gets up to 2GB in its peak. Here is a sample from stackimpact:

Is it normal that the communication requires that much ram? Could it be a problem with the bencode serialisation?

Could not find it anywhere in code, also.
Add version string and also -v command line switch.

Thanks for the tool! It works pretty nicely, but for some torrents the client is getting stuck with the following messages in the log:

go-libutp: 2019/11/18 19:47:45 callbacks.go:75: error sending packet: write udp6 [::]:50007->[2403:6200:8917:deec:7850:c7ec:20c8:e139]:34515: sendto: cannot assign requested address

I'm guessing this is because my host doesn't support ipv6. Could it be possible to disable ipv6 support entirely so that Confluence doesn't waste time on it?

It's printed when start up, any details about blocklist?

2021/01/12 07:59:22 main.go:70: open packed-blocklist: no such file or directory

Can anyone please tell how to use this? I have installed confluence with docker build command but don't know how to add torrent or magnet link and stream them

The only way to select a specific file is by its path, example:

vlc 'http://localhost:8080/data?magnet=...&path=Sintel.mp4'

Using regexp it can happen like that:

vlc 'http://localhost:8080/data?magnet=...&match=\.mp4$'

In that way, the client can create a regexp that defines its interest and simplify your flow without needing to:

1. Fetch the metainfo first.
2. Unmarshall the response.
3. Match the right file path.

This can be useful for the context of video media which usually only one video file is shared per magnet. Sample of a generic regexp that match the most popular video formats: \.(avi|mp4|mkv|webm)$.

Caveat:
Following the example, maybe a magnet points to many 'mp4's files, in that case, to maintain an easy algorithm correctness I suggest to return the first match.

Wildcard alternative:

Despite my last example, regex syntax can be really creepy, I prefer the unix shell wildcard notation that is:

vlc 'http://localhost:8080/data?magnet=...&match=*.{mp4,mkv}'

But I don't think that Golang provides a built-in support to this.

I've tried adding a torrent file but get error:

$ curl --data "@/home/porjo/Downloads/Fedora-SoaS-Live-x86_64-25.torrent" localhost:8080/metainfo?ih=1428ef237d033dc3c22410d0d4e6a565d1500ca4
error decoding body: bencode: syntax error (offset: 28406): unexpected EOF

I checked the torrent file and it is good. Could you provide an example for adding a torrent file? Also could add support for magnet links? Thanks

I would like to add trackers from magnet URIs without knowing info bytes.

It's panic after these steps. But can't be reproduced because of the data cached, maybe.

1 http://192.168.3.6:9982/info?ih=08ada5a7a6183aae1e09d831df6748d566095a10

2 http://192.168.3.6:9982/data?magnet=magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10&path=Sintel.mp4

3 http://192.168.3.6:9982/fileState?ih=08ada5a7a6183aae1e09d831df6748d566095a10&path=Sintel.mp4

root@ubuntu:/usr/local/go/bin# ./confluence -addr=:9982
2021/01/11 01:06:30 main.go:64: open packed-blocklist: no such file or directory
2021/01/11 01:06:30 main.go:204: serving http at [::]:9982
2021-01-11T01:06:36+0000 NONE  client.go:368: dht server on [::]:50007 completed bootstrap (dht.TraversalStats{NumAddrsTried:9, NumResponses:0})
2021/01/11 01:06:54 server.go:3061: http: panic serving 192.168.3.26:53029: runtime error: invalid memory address or nil pointer dereference
goroutine 560 [running]:
net/http.(*conn).serve.func1(0xc0000b9ae0)
	/usr/local/go/src/net/http/server.go:1772 +0x139
panic(0xd3da20, 0x1556630)
	/usr/local/go/src/runtime/panic.go:973 +0x3e3
github.com/anacrolix/torrent/metainfo.(*MetaInfo).UpvertedAnnounceList(0x0, 0xc000779f20, 0x30, 0x31d8091eae3a18a6)
	/root/workspace/pkg/mod/github.com/anacrolix/[email protected]/metainfo/metainfo.go:83 +0x26
main.mainErr.func4(0xc00030d400, 0x0)
	/root/workspace/pkg/mod/github.com/anacrolix/[email protected]/main.go:215 +0xa1
github.com/anacrolix/confluence/confluence.(*Handler).withTorrentContext.func1(0xf7f500, 0xc000390380, 0xc0002b0200)
	/root/workspace/pkg/mod/github.com/anacrolix/[email protected]/confluence/middlewares.go:78 +0x32f
net/http.HandlerFunc.ServeHTTP(0xc000260b20, 0xf7f500, 0xc000390380, 0xc0002b0200)
	/usr/local/go/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00020e658, 0xf7f500, 0xc000390380, 0xc0002b0200)
	/usr/local/go/src/net/http/server.go:2387 +0x1a5
github.com/anacrolix/confluence/confluence.(*Handler).ServeHTTP(0xc00020e630, 0xf7f500, 0xc000390380, 0xc0002b0200)
	/root/workspace/pkg/mod/github.com/anacrolix/[email protected]/confluence/handler.go:28 +0x5d
net/http.serverHandler.ServeHTTP(0xc000390000, 0xf7f500, 0xc000390380, 0xc0002b0200)
	/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0000b9ae0, 0xf820c0, 0xc000098580)
	/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c
^@^@

Currently, I want to test it, seed thousands of exist files and torrents.
I saw lib torrent supported it: anacrolix/torrent#138.
The problem is ./cmd/torrent don't detect exist file and will download it, I read anacrolix/torrent#374, but it can't do in command line?

E/Go: panic: page 4 already freed
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: panic: sync: Unlock of unlocked RWMutex
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: goroutine 10 [running]:
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: panic(0x7f9584cb20, 0x44202702d0)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/src/runtime/panic.go:500 +0x390
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: sync.(*RWMutex).Unlock(0x4420238130)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/src/sync/rwmutex.go:118 +0xbc
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/sync.(*RWMutex).Unlock(0x4420238000)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/sync/rwmutex.go:26 +0x68
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: panic(0x7f9584cb20, 0x44202702a0)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/src/runtime/panic.go:458 +0x4c0
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*freelist).free(0x4420165f20, 0x2b, 0x7f94080000)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/freelist.go:121 +0x280
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*node).spill(0x44204fe540, 0x0, 0x0)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/node.go:363 +0x24c
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*Bucket).spill(0x4420298480, 0x0, 0x0)
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/bucket.go:570 +0x154
06-14 01:06:50.718 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*Bucket).spill(0x44201240f8, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/bucket.go:537 +0x804
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*Tx).Commit(0x44201240e0, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/tx.go:163 +0x19c
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/boltdb/bolt.(*DB).Update(0x4420232000, 0x442003fa28, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/boltdb/bolt/db.go:605 +0x12c
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent/storage.(*boltPieceCompletion).Set(0x442016a130, 0x4952003b839e81b, 0x49331a0845795f8f, 0xa1262123, 0x1, 0x442000c000, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/storage/boltPieceCompletion.go:70 +0xa8
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent/storage.(*fileStoragePiece).MarkNotComplete(0x44202983c0, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/storage/file_storage_piece.go:49 +0x11c
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent/storage.Piece.ReadAt(0x7f95add280, 0x44202983c0, 0x4420100620, 0x1, 0x44201fa000, 0x8000, 0x8000, 0x0, 0x0, 0x7f95ad5a20, ...)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/storage/wrappers.go:79 +0x3dc
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent/storage.(*Piece).ReadAt(0x4420168520, 0x44201fa000, 0x8000, 0x8000, 0x0, 0x44201fa000, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: :47 +0xf4
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: io.(*SectionReader).Read(0x44202365d0, 0x44201fa000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/src/io/io.go:463 +0xd0
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: io.copyBuffer(0x7f91f39000, 0x44204fe460, 0x7f95ad6120, 0x44202365d0, 0x44201fa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/src/io/io.go:390 +0x1b0
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: io.Copy(0x7f91f39000, 0x44204fe460, 0x7f95ad6120, 0x44202365d0, 0x44204fe460, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/src/io/io.go:360 +0x54
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent.(*Torrent).hashPiece(0x44200c0c80, 0x1, 0x0, 0x0, 0x0)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/torrent.go:630 +0x274
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent.(*Torrent).verifyPiece(0x44200c0c80, 0x1)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/torrent.go:1486 +0x188
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: github.com/anacrolix/torrent.(*Torrent).setInfoBytes.func1(0x44200c0c80)
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/torrent.go:289 +0x44
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: created by github.com/anacrolix/torrent.(*Torrent).setInfoBytes
06-14 01:06:50.719 9996-0/com.shwifty.tex E/Go: D:/Go/projects/src/github.com/anacrolix/torrent/torrent.go:291 +0x844