Git Product home page Git Product logo

ctrader-backtest's Introduction

Read tdbc34 backtest files with Windows WSL

This Python library deserialize tdbc34 cache files from your cTrader Backtesting Cache directory.

Installation

pip install git+https://github.com/casperm/ctrader-backtest.git

Import library

from tdbc34 import CacheSerializer as cache

List all available cache files.

Run DescribeAvailableCacheData function use glob regex command to match the cache location. Parameter: env Options: wsl (default) - Windows Subsystem for Linux

cache.DescribeAvailableCacheData()[-4::]
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Ticks - has zero files
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Ticks - has zero files


[{'account': 'demo_456789',
  'instrument': 'USDCAD',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCAD/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDCAD',
  'type': 'Ticks',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCAD/Ticks',
  'from': datetime.datetime(2013, 7, 22, 0, 0),
  'to': datetime.datetime(2019, 9, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDCHF',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDJPY',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)}]

Run DescribeAvailableCacheDataInDataFrame function to list cache location in Pandas DataFrame

cache.DescribeAvailableCacheDataInDataFrame()
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Ticks - has zero files
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Ticks - has zero files
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } 
.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}
</style>
account instrument type path from to
0 demo_12345678 EURUSD Minute /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2019-12-28 2019-12-31
1 demo_12345678 EURUSD Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2019-12-31 2020-01-13
2 demo_456789 AUDJPY Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2011-01-01 2013-07-01
3 demo_456789 AUDJPY Minute /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2011-02-04 2020-08-01
4 demo_456789 AUDJPY Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-22 2020-08-01
... ... ... ... ... ... ...
61 demo_456789 US500 Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2015-01-23 2022-10-04
62 demo_456789 USDCAD Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01
63 demo_456789 USDCAD Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-22 2018-10-23
64 demo_456789 USDCHF Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01
65 demo_456789 USDJPY Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01

66 rows × 6 columns

Load Cached files

The GetBackTestData takes three parameters (cacheFiles, start_date, end_date). And returns PyArrow datatable.

%%time
_cacheFiles = "/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/US500/Ticks"
US500Ticks = cache.GetBackTestData(_cacheFiles,"2021.01.01","2021.03.01")
CPU times: user 2.47 s, sys: 45.4 ms, total: 2.51 s
Wall time: 3.3 s

Query PyArrow datatable with SQL (DuckDB OLAP Engine)

import duckdb
import pandas as pd
import sqlalchemy
# No need to import duckdb_engine
#  SQLAlchemy will auto-detect the driver needed based on your connection string!

# Import ipython-sql Jupyter extension to create SQL cells
%load_ext sql
%sql duckdb:///:memory:
%%sql
SELECT
    utctime,
    ask, bid,
    (ask + bid)/2.0 as mid,
    utctime + INTERVAL 91 second as WindowEnd,
    last(bid) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowbid,
    last(ask) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowask
FROM US500Ticks
ORDER BY utctime 
LIMIT 5
 * duckdb:///:memory:
Done.
utctime ask bid mid WindowEnd Windowbid Windowask
2021-01-03 23:00:10.072000 3758.100000 3757.500000 3757.8 2021-01-03 23:01:41.072000 3758.300000 3758.900000
2021-01-03 23:00:10.486000 3758.500000 3757.900000 3758.2 2021-01-03 23:01:41.486000 3757.900000 3758.500000
2021-01-03 23:00:10.885000 3758.600000 3758.000000 3758.3 2021-01-03 23:01:41.885000 3757.900000 3758.500000
2021-01-03 23:00:11.091000 3758.900000 3758.300000 3758.6 2021-01-03 23:01:42.091000 3757.900000 3758.500000
2021-01-03 23:00:11.686000 3759.000000 3758.400000 3758.7 2021-01-03 23:01:42.686000 3757.600000 3758.200000
2021-01-03 23:00:12.113000 3759.100000 3758.500000 3758.8 2021-01-03 23:01:43.113000 3757.400000 3758.000000
2021-01-03 23:00:12.487000 3759.400000 3758.800000 3759.1 2021-01-03 23:01:43.487000 3757.400000 3758.000000
2021-01-03 23:00:13.685000 3759.200000 3758.600000 3758.9 2021-01-03 23:01:44.685000 3757.400000 3758.000000
2021-01-03 23:00:13.887000 3760.100000 3759.500000 3759.8 2021-01-03 23:01:44.887000 3757.400000 3758.000000
2021-01-03 23:00:14.884000 3759.600000 3759.000000 3759.3 2021-01-03 23:01:45.884000 3757.000000 3757.600000 
%%time
_cacheFiles = "/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/NZDUSD/Ticks"
NZDUSDTicks = cache.GetBackTestData(_cacheFiles,"2021.01.01","2021.03.01")
CPU times: user 5.85 s, sys: 177 ms, total: 6.03 s
Wall time: 7.21 s

%%sql
SELECT
    utctime,
    ask, bid,
    (ask + bid)/2.0 as mid,
    utctime + INTERVAL 91 second as WindowEnd,
    last(bid) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowbid,
    last(ask) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowask
FROM NZDUSDTicks
ORDER BY utctime 
LIMIT 10
 * duckdb:///:memory:
Done.
utctime ask bid mid WindowEnd Windowbid Windowask
2021-01-03 22:01:00.004000 0.719410 0.719080 0.719245 2021-01-03 22:02:31.004000 0.719010 None
2021-01-03 22:01:01.105000 None 0.719100 None 2021-01-03 22:02:32.105000 0.719010 None
2021-01-03 22:01:01.909000 None 0.719080 None 2021-01-03 22:02:32.909000 0.719010 None
2021-01-03 22:01:02.429000 None 0.719070 None 2021-01-03 22:02:33.429000 0.719010 None
2021-01-03 22:01:02.954000 None 0.719050 None 2021-01-03 22:02:33.954000 0.719010 None
2021-01-03 22:01:03.307000 None 0.719030 None 2021-01-03 22:02:34.307000 0.719010 None
2021-01-03 22:01:03.548000 None 0.719010 None 2021-01-03 22:02:34.548000 0.719010 None
2021-01-03 22:03:02.290000 0.719350 0.719060 0.719205 2021-01-03 22:04:33.290000 0.718970 None
2021-01-03 22:03:18.453000 0.719410 0.719010 0.71921 2021-01-03 22:04:49.453000 0.718990 None
2021-01-03 22:03:31.321000 0.719240 None None 2021-01-03 22:05:02.321000 0.718970 None

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.