Git Product home page Git Product logo

chia-plotter's Introduction

chia-plotter (pipelined multi-threaded)

This is a new implementation of a chia plotter which is desinged as a processing pipeline, similar to how GPUs work, only the "cores" are normal software CPU threads.

As a result this plotter is able to fully max out any storage device's bandwidth, simply by increasing the number of "cores", ie. threads.

Usage

For <poolkey> and <farmerkey> see output of `chia keys show`.
<tmpdir> needs about 220 GiB space, it will handle about 25% of all writes. (Examples: './', '/mnt/tmp/')
<tmpdir2> needs about 110 GiB space and ideally is a RAM drive, it will handle about 75% of all writes.

Usage:
  chia_plot [OPTION...]

  -n, --count arg      Number of plots to create (default = 1, -1 = infinite)
  -r, --threads arg    Number of threads (default = 4)
  -u, --buckets arg    Number of buckets (default = 128)
  -t, --tmpdir arg     Temporary directory, needs ~220 GiB (default = $PWD)
  -2, --tmpdir2 arg    Temporary directory 2, needs ~110 GiB [RAM] (default = <tmpdir>)
  -d, --finaldir arg   Final directory (default = <tmpdir>)
  -p, --poolkey arg    Pool Public Key (48 bytes)
  -f, --farmerkey arg  Farmer Public Key (48 bytes)
      --help           Print help

Make sure to crank up <threads> if you have plenty of cores, the default is 4. Depending on the phase more threads will be launched, the setting is just a multiplier.

RAM usage depends on <threads> and <buckets>. With default <buckets> and 4 threads it's ~2GB total, with 16 threads it's ~6GB total.

How to Support

XCH: xch1w5c2vv5ak08pczeph7tp5xmkl5762pdf3pyjkg9z4ks4ed55j3psgay0zh

I developed this on my own time, even though I already filled all my HDDs (~50 TiB) with the official (slow) plotter.

Results

On a dual Xeon(R) [email protected] R720 with 256GB RAM and a 3x800GB SATA SSD RAID0, using a 110G tmpfs for <tmpdir2>:

Number of Threads: 16
Number of Sort Buckets: 2^7 (128)
Working Directory:   /mnt/tmp3/chia/tmp/
Working Directory 2: /mnt/tmp3/chia/tmp/ram/
[P1] Table 1 took 18.2322 sec
[P1] Table 2 took 152.806 sec, found 4294955136 matches
[P1] Table 3 took 181.893 sec, found 4295004058 matches
[P1] Lost 36189 matches due to 32-bit overflow.
[P1] Table 4 took 207.631 sec, found 4295076024 matches
[P1] Lost 108422 matches due to 32-bit overflow.
[P1] Table 5 took 210.324 sec, found 4295073698 matches
[P1] Lost 106662 matches due to 32-bit overflow.
[P1] Table 6 took 205.312 sec, found 4294986463 matches
[P1] Lost 19157 matches due to 32-bit overflow.
[P1] Table 7 took 166.982 sec, found 4294910985 matches
Phase 1 took 1143.2 sec
[P2] max_table_size = 4295076024
[P2] Table 7 scan took 16.6775 sec
[P2] Table 7 rewrite took 40.4863 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 47.6553 sec
[P2] Table 6 rewrite took 66.2653 sec, dropped 581270438 entries (13.5337 %)
[P2] Table 5 scan took 45.4674 sec
[P2] Table 5 rewrite took 67.3295 sec, dropped 762045018 entries (17.7423 %)
[P2] Table 4 scan took 44.4048 sec
[P2] Table 4 rewrite took 62.1307 sec, dropped 828943971 entries (19.2999 %)
[P2] Table 3 scan took 45.1361 sec
[P2] Table 3 rewrite took 61.2604 sec, dropped 855078559 entries (19.9087 %)
[P2] Table 2 scan took 46.5209 sec
[P2] Table 2 rewrite took 61.3043 sec, dropped 865523252 entries (20.1521 %)
Phase 2 took 635.101 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 80.6498 sec, wrote 3429431884 right entries
[P3-2] Table 2 took 74.787 sec, wrote 3429431884 left entries, 3429431884 final
[P3-1] Table 3 took 70.8236 sec, wrote 3439925499 right entries
[P3-2] Table 3 took 77.8787 sec, wrote 3439925499 left entries, 3439925499 final
[P3-1] Table 4 took 71.894 sec, wrote 3466132053 right entries
[P3-2] Table 4 took 76.3172 sec, wrote 3466132053 left entries, 3466132053 final
[P3-1] Table 5 took 72.6806 sec, wrote 3533028680 right entries
[P3-2] Table 5 took 81.5741 sec, wrote 3533028680 left entries, 3533028680 final
[P3-1] Table 6 took 77.1653 sec, wrote 3713716025 right entries
[P3-2] Table 6 took 85.9674 sec, wrote 3713716025 left entries, 3713716025 final
[P3-1] Table 7 took 75.5146 sec, wrote 4294910985 right entries
[P3-2] Table 7 took 93.9135 sec, wrote 4294910985 left entries, 4294910985 final
Phase 3 took 946.026 sec, wrote 21877145126 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 79.6722 sec, final plot size is 108835267927 bytes
Total plot creation time was 2804.06 sec

How to Verify

To make sure the plots are valid you can use the ProofOfSpace tool from chiapos:

git clone https://github.com/Chia-Network/chiapos.git
cd chiapos && mkdir build && cd build && cmake .. && make -j8
./ProofOfSpace check -f plot-k32-???.plot [num_iterations]

Future Plans

I do have some history with GPU mining, back in 2014 I was the first to open source a XPM GPU miner, which was about 40x more efficient than the CPU miner. See my other repos.

As such, it's only a matter of time until I'll add OpenCL support to speed up the plotter even more, keeping most of the load off the CPUs.

Dependencies

  • cmake (>=3.14)
  • libgmp3-dev
  • libsodium-dev

Install

Ubuntu 20.04

sudo apt install  -y libsodium-dev  libgmp3-dev  cmake g++ git
git clone https://github.com/madMAx43v3r/chia-plotter.git 
cd  chia-plotter
git submodule update --init
./make_devel.sh

The binaries will end up in build/, you can copy them elsewhere freely (on the same machine, or similar OS).

Known Issues

  • Doesn't compile with gcc-11, use a lower version.
  • Needs at least cmake 3.14 (because of bls-signatures)

How to install latest cmake on 18.04

https://askubuntu.com/questions/1203635/installing-latest-cmake-on-ubuntu-18-04-3-lts-run-via-wsl-openssl-error

chia-plotter's People

Contributors

fermosan avatar gtanadam avatar madmax43v3r avatar stotiks avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.