Improve start-up efficiency when hashing with file-based storage backends about torrent HOT 16 CLOSED

Thanks for the report. I believe the cause is the way I offload writes to disk which can be too slow for a fast torrent. If you're using a wrapper, could you try the mmap backed data store as an alternative? I'll address the too many open files issue soon. Thanks.

from torrent.

If you're using a wrapper, could you try the mmap backed data store as an alternative?

Can you explain?

from torrent.

In your code at https://github.com/zhulik/go_torrent_bot/blob/master/main.go#L34, you can set TorrentDataOpener to the mmap.TorrentData backend, an example is here in the tests: https://github.com/anacrolix/torrent/blob/master/fs/torrentfs_test.go#L206. This avoids using Open/Close and file handle overhead. Let me know if it's still a problem with this.

from torrent.

Hi, i have updated torrent package, set mmap.TorrentData as you told me.
I see, that memory usage still high, htop shows about 500MB, stable.
CPU usage is high too: 100% of first core + 30-50% of second.

Positive changes: i don't see large numbers of threads, just 7, but on 80% of downloading it was increased to 32. Dowloaded files are valid, no too many open files errors.

Negative:
After downloading, i still have 100% cpu usage by kswap0 process, it is common problem in linux, but no one of found solutions doesn't work for me(maybe problem in kernel, i have old 4.1.6 on target machine, mayby your mmap realization does something wrong with it)

After killing downloader kswap0 process calmed down

All of the above is true for big torrent, about 15Gb(one season of popular animated series, 20 mkv file, each about 600mb).

For small torrents is see only high CPU usage while downloading

from torrent.

That log message is not originating from the torrent package. Perhaps
something else is going on.
On 25 Dec 2015 8:53 am, "Gleb Sinyavsky" [email protected] wrote:

Hi, i hade updated torrent package, set mmap.TorrentData as you told me.
I see, that memory usage still high, htop shows about 500MB, stable.
CPU usage is high too: 100% of first core + 30-50% of second.

Positive changes: i don't see large numbers of threads, just 7, but on 80%
of downloading it was increased to 32. Dowloaded files are valid, no too
many open files errors.

Negative:
2015/12/25 08:25:58 Failed to get updates, retrying in 3 seconds... in
logs after downloading

After downloading, i still have 100% cpu usage by kswap0 process, it is
common problem in linux, but no one of founded solutions doesn't work for
me(maybe problem in kernel, i have old 4.1.6 on target machine, mayby your
mmap realization does something wrong with it)

After killing downloader kswap0 process calmed down

—
Reply to this email directly or view it on GitHub
#50 (comment).

from torrent.

Sorry, you answered on old version of comment. I have updated it. Please, read it from web.

from torrent.

Here is my cpu profile. I have no experience in reading go's profiles, but i suppose, that crypto/sha1.block is slow. Am i right?
cpu.pdf

from torrent.

Thanks. How large is the torrent you're downloading? Are you using the file or mmap data backend? What OS/architecture are you using?

from torrent.

I was downloading many torrents about 15-20Gb, all of them has the same client behavior.

Client initialization you can find here, mmap storage. OS: ArchLinux latest, x86_64, 100Mbit internet connection, transmission shows me ~5-7 Mb/s speed

from torrent.

I just configured newrelic monitoring and then download 20Gb torrent. There is graphs

1400 active goroutins while downloading is really great amont. Scheduling of them is not free. Maybe, decreasing it's amount is a good idea?

There is only one active download. What happens if i add many big torrents?

from torrent.

So the cause is that on startup, torrent scans pieces from all files for the mmap and file backends rehashes them all. I moved onto using the pieces backend for my production purposes, and so those file-based backends haven't had enough attention. Some state between restarts is needed. On slow disks (I'm guessing spinning media)/CPU, and for large torrent, the rescan can be very expensive and take a while before the torrent can actually do anything else due to being starved for CPU. This can be confirmed by trying with small torrents and seeing they finish hashing first, or waiting out the hashing, and check that CPU drops and download speed picks up.

You can use the Client.WriteStatus to get a read out of what the torrent client is doing, you should see lots of pieces in the 'H' state, like 123H means 123 pieces are being hashed, when the CPU is up.

Let me know, and if you have any suggestions for where to maintain that state. I'm currently considering just a sqlite3 file in the directory that maps pieces to whether they've been checked before or something. Maybe like '.torrent.db'.

from torrent.

Sorry to piggyback on the discussion, but it seems a bit ontopic.Can you give a small example on how to use the pieces backend?

from torrent.

@0x4139 I put an example in the other issue.

from torrent.

@zhulik Any update on your usage of this? Are you still having issues?

from torrent.

I've started working on this.

from torrent.

This was fixed in d80d0db. There's a more recent bug fix to it in 11a53fa.

from torrent.