Unclear how to run this tool with Gekko in Docker container. about gekko-backtesttool HOT 19 CLOSED

I have no idea how Docker works: D You can run Gekko using the system command (ie from the bash level)? I mean node gekko -c -b or sth similar?

Or, if you have the opportunity to copy the file to cointainera without rebuilding? For example, copying binance_0.1.db to gekko / history?

from gekko-backtesttool.

Gekko is in /usr/src/app directory in container. Try:

1. docker cp backtest.pl backtest-config.pl <container>:/usr/src/app
2. Try a) if BT doesnt start and print error about missed modules then try b)
   a) docker exec -ti --user root <container-id> /usr/local/bin/perl /usr/src/app/backtest.pl
   b) docker exec -ti --user root <container-id> /bin/bash and next do https://github.com/xFFFFF/Gekko-BacktestTool#open-source-other-unix-like-os

Let me know do it work or You could prepare short tutorial to README and add pull request :)

from gekko-backtesttool.

Thank you! I've tried the whole evening, building a container with the dependencies being installed by the dockerfile but it didn't work out due to dependency issues (dbd csv). I'll try your suggestion as soon as I have time and report back to you! I managed to build a script which combines all strategies and gekko in one container automatically, if only I was able to include the backtesting tool by using a dockerfile... to be continued!

Btw, I'm building for raspbian stretch (running on raspberry pi).

from gekko-backtesttool.

You can try run binary version in docker without installing dependencies. What issue you had with DBD::CSV?

from gekko-backtesttool.

The binary version was compiled for amd64, while the Raspberry Pi has an ARM processor, so I couldn't find the right binaries. I will give you the details of the error once I'm back at my pc, don't know off the top of my head. I'll also post the dockerfile (which is a modification of the gekko Dockerfile, it now includes a RUN cpan install to install the dependencies in the container). Probably later this day. It would really be nice to have them all running in one container.

from gekko-backtesttool.

If the installation in the cointainer will be problematic - if you need to install perl, compilers and others, maybe you will like this solution more: install dependencies on localhost Raspberry (Perl should be installed by default), compile backtest.pl, and then you will transfer the compiled version to the container, it should work even if you do not have Perl in the cointerator.

If you want to "compile" on your own, just cpan install PAR::Packer and then the command: pp -M Time::Elapsed::Lang::* backtest.pl -o backtest and its all. Of course, if everything goes ok.

from gekko-backtesttool.

When executing "perl backtest.pl" in the docker container, I get the following error:

Can't locate DBD/CSV.pm in @inc (you may need to install the DBD::CSV module) (@inc contains: /etc/perl /usr/local/lib/arm-linux-gnueabihf/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/arm-linux-gnueabihf/perl5/5.20 /usr/share/perl5 /usr/lib/arm-linux-gnueabihf/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at backtest.pl line 21.
BEGIN failed--compilation aborted at backtest.pl line 21.

I did include the following line in the Dockerfile (which builds the container):
RUN cpan install Parallel::ForkManager Time::Elapsed Getopt::Long List::MoreUtils File::chdir Statistics::Basic DBI DBD::SQLite JSON::XS TOML File::Basename File::Find::Wanted Template LWP::UserAgent LWP::Protocol::https Set::CrossProduct DBD::CSV Text::Table File::Copy

so all dependencies should be installed. The *.pl files are present in the /usr/src/app/ folder. Perl version in the container is:

This is perl 5, version 20, subversion 2 (v5.20.2) built for arm-linux-gnueabihf-thread-multi-64int

from gekko-backtesttool.

Making progress! After installing all packages manually in the container, it starts to work. I receive the following error though:

USDT-BTC Backtest is failed.
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196.

from gekko-backtesttool.

mkdir /usr/src/app/tmp.
Are You using latest version? On BT start should create tmp directory for temp configs and csv file.

I dont know why but more people had problems with missed DBD::CSV. Looking like wasnt installed at first try.

from gekko-backtesttool.

That's very strange, I create the tmp directory in /usr/src/app and double checked if it exists. I still get the same error, but noticed that the directory is being removed by the script? I do use the latest version of all repositories (gekko, strategies and backtesttool).

[2018-05-19 13:26:54] All jobs are done. Elapsed time: zero seconds

root@bf7d704bb99b:/usr/src/app# USDT-BTC Backtest is failed.
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196.
USDT-BTC Backtest is failed.
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196.
USDT-BTC Backtest is failed.
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196.
USDT-BTC Backtest is failed.
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196.

from gekko-backtesttool.

You can comment last line in BT rmtree 'tmp' to prevent removing by BT.
I havent idea why perl mkdir not work. Have You this line on begin of code: mkdir 'tmp' unless -d 'tmp';?

from gekko-backtesttool.

Yes, I have this line:

my $configfile = "./backtest-config.pl";
mkdir 'tmp' unless -d 'tmp';

Permission issue?

from gekko-backtesttool.

Maybe, it would be weird. Any app can use perl to create directory. BT has disabled warnings. Can You try these lines in docker?
https://gist.github.com/xFFFFF/2705dc49b68cc639232c56ce1aadf5b7
Its two methods, first is system enviroment call, second is perl mkdir. I cant implement system call to BT because will not work on Windows.

BT creating files too. Does exist? In tmp directory and in main gekko dir backtest.dat (I must move it to tmp).

from gekko-backtesttool.

So I fixed the issue with the perl dependencies by using cpanm instead of cpan in the dockerfile. I'll post the result when I'm back at my pc. Only issue now is the "tmp" directory. I'll try your gist again when I'm back at my pc and when I discovered how to install the dependency (see comment at gist). Thank you for your dedicated help!

from gekko-backtesttool.

Making progress again. Tool seems to be running fine now, except that it says that the database is locked. This seems to be happening once a manual data import is done in the GUI. Before that it seems to work fine. Maybe an issue for Gekko.

perl backtest.pl -i

Updating Gekko's kraken market data...
Done writing Kraken market data
Import 1 pairs: kraken:EUR:XRP.
EUR:XRP (2018-01-01 00:00:00 - 2018-01-02 00:00:00) is started at 2018-01-02 00:00:00...
events.js:183
throw er; // Unhandled 'error' event
^

Error: SQLITE_BUSY: database is locked
Importing kraken EUR-XRP is failed:
Had 1 errors on 1 jobs. Elapsed time: 2 seconds

from gekko-backtesttool.

You have right with import: https://github.com/xFFFFF/Gekko-BacktestTool/wiki/Troubleshooting#error-sqlite_busy-database-is-locked

So You making progress or all is working?

from gekko-backtesttool.

This issue can be closed, I will post results when I have a complete script which merges all relevant repositories into one directory and which alters the Dockerfile in order to build a container. Right now the solution is to:

• Clone gekko repo
• Clone Gekko-BacktestTool repo
• (optionally) Clone Gekko-Strategies repo
• (optionally) Clone gab0/gekko-extra-indicators
• Install all repositories into the gekko repo according to their respective install instructions.

Change the Dockerfile of the gekko repo to (to install cpanminus and the perl dependencies):

=======================================================
FROM node:8

ENV HOST localhost
ENV PORT 3000

# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Install GYP dependencies globally, will be used to code build other dependencies
RUN npm install -g --production node-gyp &&
npm cache clean --force

# Install app dependencies
COPY package.json /usr/src/app
RUN npm install --production &&
npm install --production [email protected] [email protected] [email protected] pg &&
npm cache clean --force

# Install cpanminus.
RUN curl -L http://cpanmin.us | perl - App::cpanminus

# Install perl dependencies for Gekko-BacktestTool.
RUN cpanm -n install Parallel::ForkManager Time::Elapsed Getopt::Long List::MoreUtils File::chdir Statistics::Basic DBI DBD::SQLite JSON::XS TOML File::Basename File::Find::Wanted Template LWP::UserAgent LWP::Protocol::https Set::CrossProduct DBD::CSV Text::Table File::Copy

# Bundle app source
COPY . /usr/src/app

EXPOSE 3000
RUN chmod +x /usr/src/app/docker-entrypoint.sh
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]

CMD [ "npm", "start" ]

=======================================================

And build your Docker container. To execute the backtesttool from within the container, use the following command on the Docker host:

docker exec -it [image ID] perl backtest.pl

When you fire up 2 containers of the same image, you automatically solve the issue of "SQLITE_BUSY: database is locked". Use one container for your trading, one container for your backtesting.

from gekko-backtesttool.

Excellently! Thanks for your work :) I will create a subpage for Docker and put your tutorial on. I also link in readme :)

from gekko-backtesttool.

Hello! I have the same issue with tmp/.
perl backtest.pl
Error in tempfile() using template tmp/tmp_configXXXXX.js: Parent directory (tmp/) does not exist at backtest.pl line 1196

I do not use docker.
Anyone can help me?

from gekko-backtesttool.