Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Files or part of package has been run:

Expected behaviour:

Actual behaviour:

Steps to reproduce:

I have some csv files that I would like to use as datasource of the format

timestamp,open,high,low,close,volume

What would I have to modify to support this?

Hi,

Great work.

I was looking for examples of market trading environments for OpenAI for an idea I had around machine learning for risk management (I don't think you can teach a computer to trade on random walk data better than you can teach a human - it's like locking both in a box and hoping they can make money) but what I am hoping you can do is teach them risk management/position sizing based on prevailing market conditions which has much more useful applications.

Hooking OpenAI into Backtrader turns out to be more than I could hope for.

Just one issue I ran into which might be worth noting for others or updating your documentation...

When running your examples they all (or at least all the ones I tried) log "No running server found." then exit.

The issue is that you haven't got an agent running. It might be worth demonstrating a basic agent. Eg:

env = ..........
env.reset()
for _ in range(1000):
env.render()
env.step(env.action_space.sample()) # take a random action

Hi @Kismuz
I need your help to clarify a few questions:

1. Currently, Unreal Stacked LSTM Strat 4_11 has better performance on actual FOREX prices than A3C Random, so just out of curiosity, I tried to run both of them with the sine wave data (test_sine_1min_period256_delta0002.csv). It turns out that A3C eventually converges while Unreal Stacked LSTM is nowhere near the convergence. Could you provide me some insights on that?

LSTM results:

A3C results:

2. I had an overflow issue with the tanh(x_sma) while feeding my own data into the gym (SPY, Dow Jones, Russel 2000, and so on). After changing the T2 value from 2000 to 1, the issue went away but I am not sure if that was a proper fix. Could you help me shed some lights on that?

3. I also got data consistency issue when passing equity data traded from 8:30am to 15:15pm instead of 24/7 like FOREX data that you used as examples. What should I do to fix this?

4. I kept getting the "[WinError 2] The system cannot find the file specified" when trying to run the A3C example in Windows.
   

5. When I tried to run the A3C with the new version of tensorflow supporting GPU in Windows, I I got the error below also:
   

Really appreciate your effort to build this awesome gym and great documentation.

Thank you in advance.

I would like to know how is computed the reward function ? Is it based on the open/low price ?
Also, what is the meaning of the action "close" ?

Hi, it is so nice that you wrote these two setting up notebook, it helps a lot, but when I was running the example/setting_up_environment_full，I got these error messages, could you help me out?

env.reset()
take_some_steps(env, 100)
render_all_modes(env)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
~/projects/btgym/btgym/envs/backtrader.py in _reset(self, state_only)
    606             try:
--> 607                 assert self.observation_space.contains(self.env_response[0])
    608 

AssertionError: 

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
<ipython-input-10-af63753eff47> in <module>()
----> 1 env.reset()
      2 take_some_steps(env, 100)
      3 render_all_modes(env)

~/.pyenv/versions/miniconda3-4.3.11/lib/python3.6/site-packages/gym/core.py in reset(self)
    102             space.
    103         """
--> 104         return self._reset()
    105 
    106     def render(self, mode='human', close=False):

~/projects/btgym/btgym/envs/backtrader.py in _reset(self, state_only)
    632                 self.log.error(msg)
    633                 self._stop_server()
--> 634                 raise AssertionError(msg)
    635 
    636             if state_only:

AssertionError: 
State observation shape/range mismatch!
Space set by env: 

price_gradients:
   Box(30, 4), low: 0.0, high: 1.0

raw_state:
   Box(30, 4), low: 1.03522, high: 1.1616

Space returned by server: 

raw_state:
   array of shape: (4, 4), low: 1.11363, high: 1.11372

price_gradients:
   array of shape: (4, 4), low: 0.3965167501352529, high: 0.6726070170682124

Full response:
{'raw_state': array([[ 1.11372,  1.11372,  1.1137 ,  1.11371],
       [ 1.1137 ,  1.1137 ,  1.11369,  1.11369],
       [ 1.11369,  1.11371,  1.11363,  1.11371],
       [ 1.1137 ,  1.11372,  1.11369,  1.11372]]), 'price_gradients': array([[ 0.44028635,  0.44028635,  0.47003595,  0.44028635],
       [ 0.45512111,  0.4850045 ,  0.39651675,  0.5       ],
       [ 0.5       ,  0.52996405,  0.5       ,  0.54487889],
       [ 0.52996405,  0.52996405,  0.67260702,  0.52996405]])}
Reward: 0.0
Done: False
Info:
{'step': 5, 'time': datetime.datetime(2016, 7, 4, 0, 7), 'action': 'hold', 'broker_message': '-', 'broker_cash': 100.0, 'broker_value': 100.0, 'drawdown': 0.0, 'max_drawdown': 0.0}

Hint: Wrong Strategy.get_state() parameters?

I have two issues while running the code, first one is when I try to run a basic environment. After performing the first action, I receive the following error :
ValueError: too many values to unpack (expected 4)
and when I print the output of env.step(action) :
"No key received:Server Control mode: received <{'action': 'hold'}>"

from btgym import BTgymEnv from policy_gradient import PolicyGradient import numpy as np env = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',) done = False while not done: action = env.action_space.sample() obs, reward, done, info = env.step(action) print(reward)

Second issue is related to the rewards, the are equal to 0 whatever the actions I choose.

I run tensorborad_monitor.ipynb and found this problem below: Please help

UnsupportedMode Traceback (most recent call last)
in ()
79 images_monitor.write(
80 feed_dict={'human': env.render('human')[None,:],
---> 81 'agent': env.render('agent')[None,:],
82 },
83 global_step=global_step + last_i['step'],

~/.local/lib/python3.5/site-packages/gym/core.py in render(self, mode, close)
147 raise error.UnsupportedMode('{} does not support rendering (requested mode: {})'.format(self, mode))
148 elif mode not in modes:
--> 149 raise error.UnsupportedMode('Unsupported rendering mode: {}. (Supported modes for {}: {})'.format(mode, self, modes))
150 return self._render(mode=mode, close=close)
151

UnsupportedMode: Unsupported rendering mode: agent. (Supported modes for : ['human', 'episode'])

Hello @Kismuz
There is something I am experiencing with the environment where as at each step when taking an action on the environment the reward is always 0. I am not sure this is a bug or I am just missing something

Here is my initialization code

env = BTgymEnv(
            filename='./btgym/examples/data/DAT_ASCII_EURUSD_M1_2010.csv',
            start_weekdays=[0, 1],
            episode_duration={'days': 2, 'hours': 0, 'minutes': 0},
            strategy=MyStrategy,
            start_00=True,
            start_cash=self.capital,
            broker_commission=0.002,
            fixed_stake=10,
            drawdown_call=30,
            state_shape={'raw_state': spaces.Box(low=1, high=2, shape=self.input_shape),'indicator_states': spaces.Box(low=-1, high=100, shape=self.input_shape)},
            port=5002,
            data_port=4800,
            verbose=1,)

Here is how it is used

done = False
    while not done:
        action = agent.act(state)
        next_state, reward, done, info = env.step(action) #The reward here is always 0 regardless of the action I take
        agent.remember(state, action, reward, next_state, done)
        state = next_state

Expected behaviour:

I expect the reward to be varying in that it should be negative or positive or zero occationally.

Actual behaviour:

The reward is always 0 even when I put all the actions within one episode to be a single one.

Please help me out here

Thanks in advance.

hi Andrew,

I got a new issue when try to run A3C algorithm in example a3c_random_on_synth_or_real_data_4_6.ipynb, I run the file in a terminal with cmd: python xxx.py, but I got the following error,have you encountered this problem before?

INFO:Env:Environment is ready.
WARNING:worker_8:AAC_8: learn_rate: 0.000100, entropy_beta: 0.038476
INFO:Env:Server started, pinging tcp://127.0.0.1:5009 ...
DEBUG:Env:Server Control mode: received <{'ctrl': 'ping!'}>
DEBUG:Env:Server sent: {'ctrl': 'send control keys: <_reset>, <_getstat>, <_render>, <_stop>.'}
DEBUG:Env:Server seems ready with response: <{'ctrl': 'send control keys: <_reset>, <_getstat>, <_render>, <_stop>.'}>
INFO:Env:Environment is ready.
WARNING:worker_9:AAC_9: learn_rate: 0.000100, entropy_beta: 0.010318
2017-12-29 02:14:33.860902: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:34.660841: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.352340: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.357352: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.362327: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.363592: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.368915: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.372948: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.377060: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.388348: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0

In the documentation (https://kismuz.github.io/btgym/intro.html#quickstart), Maximum environment flexibility section, MyCerebro.addstrategy(BTgymStrategy,....

However, BTgymStrategy is not defined here. Do we need to define it first?

Hello,

When I run the UNREAL example I got the following output.

/home/jsalvado/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
</home/jsalvado/tmp/test_gym_unreal> already exists. Override[y/n]? y
WARNING:Launcher:Files in </home/jsalvado/tmp/test_gym_unreal> purged.
2017-11-27 16:52:54.666375: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:54.670453114 18319 ev_epoll1_linux.c:1051] grpc epoll fd: 7
2017-11-27 16:52:54.671044: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:54.671596938 18320 ev_epoll1_linux.c:1051] grpc epoll fd: 8
2017-11-27 16:52:54.676864: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:12230}
2017-11-27 16:52:54.676891: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> 127.0.0.1:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:54.677761: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:54.677801: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12231, 1 -> 127.0.0.1:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:54.677844: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12230
2017-11-27 16:52:54.679672: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12231
Press Ctrl-C or [Kernel]->[Interrupt] to stop training and close launcher.
2017-11-27 16:52:59.683070: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:59.683829609 18359 ev_epoll1_linux.c:1051] grpc epoll fd: 9
2017-11-27 16:52:59.686654: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:59.687214727 18360 ev_epoll1_linux.c:1051] grpc epoll fd: 10
2017-11-27 16:52:59.689904: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:59.689941: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> localhost:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:59.690832: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12232
2017-11-27 16:52:59.693367: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:59.693405: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> 127.0.0.1:12232, 2 -> localhost:12233}
2017-11-27 16:52:59.694368: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12233
2017-11-27 16:53:04.660194: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session f67f491b8dcaa755 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:0/cpu:0" inter_op_parallelism_threads: 2
2017-11-27 16:53:09.148756: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session dd584888bb6349a4 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:1/cpu:0" inter_op_parallelism_threads: 2
2017-11-27 16:53:09.294430: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session 785d00122230e0e1 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:2/cpu:0" inter_op_parallelism_threads: 2
WARNING:worker_1:worker_1: started training at step: 0
WARNING:worker_2:worker_2: started training at step: 0
WARNING:worker_0:worker_0: started training at step: 0
WARNING:Env:Data_master reset() called prior to reset_data() with [possibly inconsistent] defaults.
WARNING:Env:Dataset not ready, waiting time left: 298 sec.
WARNING:Env:Dataset not ready, waiting time left: 298 sec.

Do you know what can be done to make it work?
Thank you very much.

João Salvado

Hi,
when I try to derivate BTgymDataset in order to use my own datasource in this way: https://www.backtrader.com/blog/posts/2015-08-11-datafeed-development/datafeed-development.html

I get btgym complaining about the NoneType being not iterable. It is trying to read a csv file even though I did override the necessary methods such as load.

Is there a use case I am missing so that I can use a non csv source of data in the same way backtrader does in the example linked above?
Thx.

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Manjaro Linux Anaconda 5.1 Python 3.6
Jupyter Notebook

Files or part of package has been run:

atari_a3c

Process DrawCerebro-2:1:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/git/btgym/btgym/rendering/plotter.py", line 75, in run
    figfilename='_tmp_btgym_render.png',
  File "/home/btgym/lib/python3.5/site-packages/backtrader/cerebro.py", line 991, in plot
    start=start, end=end, use=use)
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 220, in plot
    self.plotdata(data, self.dplotsover[data])
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 634, in plotdata
    data, opens, highs, lows, closes, volumes, vollabel)
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 580, in plotvolume
    ax.set_ylim(0, volylim, auto=True)
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 3226, in set_ylim
    top = self._validate_converted_limits(top, self.convert_yunits)
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2836, in _validate_converted_limits
    raise ValueError("Axis limits cannot be NaN or Inf")
ValueError: Axis limits cannot be NaN or Inf

let self.volume = False then,

Process DrawCerebro-2:1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/git/btgym/btgym/rendering/plotter.py", line 78, in run
    rgb_string = fig.canvas.tostring_rgb()
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 460, in tostring_rgb
    return self.renderer.tostring_rgb()
AttributeError: 'FigureCanvasAgg' object has no attribute 'renderer'

When I ran the following example code：

from btgym import BTgymEnv
MyEnvironment = BTgymEnv(filename='/home/Code/btgym-master/examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
episode_duration={'days': 2, 'hours': 23, 'minutes': 55},
drawdown_call=50,
state_shape=(4,20),
port=5555,
verbose=1,
)

[@-pc btgym-master]$ cd /home/Code/btgym-master ; env "PYTHONIOENCODING=UTF-8" "PYTHONUNBUFFERED=1" python /home/.vscode/extensions/ms-python.python-2018.3.1/pythonFiles/PythonTools/visualstudio_py_launcher.py /home/Code/btgym-master 33543 34806ad9-833a-4524-8cd6-18ca4aa74f14 RedirectOutput,RedirectOutput /home/Code/btgym-master/examples/test.py
[2018-04-02 03:29:58.845135] INFO: BTgymAPIshell_0: Base Dataset class used.
[2018-04-02 03:29:58.845766] INFO: BTgymAPIshell_0: Connecting data_server...
[2018-04-02 03:29:58.936939] INFO: BTgymDataServer_0: PID: 6136
[2018-04-02 03:29:59.969214] INFO: SimpleDataSet_0: Loaded 372678 records from </home//Code/btgym-master/examples/data/DAT_ASCII_EURUSD_M1_2016.csv>.
[2018-04-02 03:30:00.145812] INFO: SimpleDataSet_0: Data summary:
open high low close volume
count 372678.000000 372678.000000 372678.000000 372678.000000 372678.0
mean 1.107109 1.107198 1.107019 1.107108 0.0
std 0.024843 0.024840 0.024847 0.024844 0.0
min 1.035250 1.035470 1.035220 1.035220 0.0
25% 1.092140 1.092230 1.092040 1.092140 0.0
50% 1.113530 1.113610 1.113450 1.113530 0.0
75% 1.124710 1.124780 1.124630 1.124710 0.0
max 1.161440 1.161600 1.160770 1.161450 0.0
[2018-04-02 03:30:00.152136] INFO: BTgymAPIshell_0: ...done.
[2018-04-02 03:30:00.154973] INFO: BTgymAPIshell_0: Base Cerebro class used. Base Strategy class used.
Traceback (most recent call last):
File "/home/liguodong/Code/btgym-master/examples/test.py", line 28, in
verbose=1,
File "/home/liguodong/Code/btgym-master/btgym/envs/backtrader.py", line 349, in init
if 'raw_state' in self.params['strategy']['state_shape'].keys():
AttributeError: 'tuple' object has no attribute 'keys'

hi Andrew,
Thanks for your great work! I wonder if I can run this code on my GPU env. I've tried modify the code 'cpu:0' to 'GPU:0' but got error saying that resource are not available.

Could you pls tell me what's the correct operation?

Tom

I know a current limitation is accept Forex 1 min (only Forex?), but my datasets are with bigger timeframes.

This is the stacktrace when timeframe is changed:

Process BTgymServer-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()

  File "/home/adrian/btgym/btgym/server.py", line 405, in run
    episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1142, in run
    runstrat = self.runstrategies(iterstrat)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1327, in runstrategies
    self.stop_writers(runstrats)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1352, in stop_writers
    datainfos['Data%d' % i] = data.getwriterinfo()

  File "/usr/local/lib/python3.5/dist-packages/backtrader/dataseries.py", line 101, in getwriterinfo
    info['Timeframe'] = TimeFrame.TName(self._timeframe)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/dataseries.py", line 57, in TName
    return cls.Names[tframe]
IndexError: list index out of range

Any idea?

First of all, thank you for this project, I really believe it has a great potential and also let me tell that I'm nothing but a newbie so excuse my question if it is a dumb one.

When I configure and start running, during some epocs it keeps giving SELL actions although there is not any holding assets. 2 questions:

• Why is the reward decreases when it sells even though there isn't anything to sell
• Can we change that in a way that the action list is based on current holdings so it doesn't try to buy when there isn't any money left or sell when there isn't anything to sell

Thanks again

@Kismuz First of all, thank you so much for creating this awesome gym. It is probably the best gym for trading that I could find on the web at the moment.

As I have read all the threads from other issues and also run the A3C Random notebook that you provided, the reward did not converge over time. I saw you have mentioned that as of now, you have not found any good solution that could lead to the convergence. I have spent sometime with the package, especially with the notebooks examples, and just thought maybe if we provided more information to the agent like technical indicators or the like, it would help improve the performance. I have tried to modify the Dev_Strat_4_6 by adding two simple moving average (50, 100) but keep getting the State observation shape/range mismatch, which I believe is due to the NAN problem since there is no data for the second indicator for the first 100 records. I could fix the problem by set the skip_frame to 100 but this is not a very data-efficient solution. Could you please help instruct me how to tackle this problem? Thank you in advance.

IMO, dockerizing is absolutely helpful to reduce environment issues, and also helpful to distributed computing.

Hi Kizmuz, First of all congratulations for this really great job!

OK, I have trained my workers, and here you are some questions:

Do you have some code to test predictions with their model?
Do you have some code to extract weights from the checkpoint file and deploy the model? If not, could you provide with some guideline instructions to do that?
How do you plan to implement the production test?

Thank you in advance.

dormat(mode) > format(mode)

raise ValueError('Unexpected render mode {}'.format(mode))

Hello first of all thank you for your contribution.
I would like to know what is your opinion concerning using A3C to predict stock market and also for a certain equity what kind of features should be taken into account? High , low, open close ?

Thank you
João Salvado

While runing the examples, I'm getting error like below

INFO:tensorflow:Starting queue runners. WARNING:worker_1:worker_1: started training at step: 120 Exception in thread Thread-4: Traceback (most recent call last): File "/home/humin/anaconda3/envs/tensorflow/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/home/humin/btgym/btgym/algorithms/runner.py", line 75, in run self._run() File "/home/humin/btgym/btgym/algorithms/runner.py", line 96, in _run self.queue.put(next(rollout_provider), timeout=600.0) File "/home/humin/btgym/btgym/algorithms/runner.py", line 238, in env_runner episode_stat = env.get_stat() # get episode statistic File "/home/humin/btgym/btgym/envs/backtrader.py", line 680, in get_stat if self._force_control_mode(): File "/home/humin/btgym/btgym/envs/backtrader.py", line 508, in _force_control_mode self.server_response = self.socket.recv_pyobj() File "/home/humin/anaconda3/envs/tensorflow/lib/python3.5/site-packages/zmq/sugar/socket.py", line 491, in recv_pyobj msg = self.recv(flags) File "zmq/backend/cython/socket.pyx", line 693, in zmq.backend.cython.socket.Socket.recv File "zmq/backend/cython/socket.pyx", line 727, in zmq.backend.cython.socket.Socket.recv File "zmq/backend/cython/socket.pyx", line 150, in zmq.backend.cython.socket._recv_copy File "zmq/backend/cython/socket.pyx", line 145, in zmq.backend.cython.socket._recv_copy File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc zmq.error.Again: Resource temporarily unavailable
I tried to reduce the number of workers but it looks irrelevant. This error is not interrupting the training process. Is this normal or should I be concerned about it?
My environment: Ubuntu 16.04. Python 3.5

Hi,

Finally got around to trying a real "deep learning" implementation against btgym and I've run up against a problem. I really don't know enough about openai gym to understand what the problem is fully but it seems to me based on the implementation I am trying and observing other implementations that observation_space.shape returns inconsistent values.

I tried modifying this basic A3C implementation: https://github.com/jaara/AI-blog/blob/master/CartPole-A3C.py

I've attached my attempt below but when I run it I get the error:

ValueError: Error when checking : expected input_1 to have 2 dimensions, but got array with shape (1, 4, 30)

When I print observation_space.shape it seems to change all the time which is expected but I believe it should always follow the same format (again I'm a bit of a newbie with this stuff so I could be wrong).

Are you able to take a look? I'll keep digging and try another implementation - I'm thinking a DQN implementation. The problem with most of the implementations for DQN or A3C is that they rely on the state being implemented as a screenshot so an array of rows and columns sometimes represented as RGB or often just 0 or 1.

Keep up the good work BTW.

# OpenGym CartPole-v0 with A3C on GPU
# -----------------------------------
#
# A3C implementation with GPU optimizer threads.
# 
# Made as part of blog series Let's make an A3C, available at
# https://jaromiru.com/2017/02/16/lets-make-an-a3c-theory/
#
# author: Jaromir Janisch, 2017

import numpy as np
import tensorflow as tf

import gym, time, random, threading
from btgym import BTgymEnv


from keras.models import *
from keras.layers import *
from keras import backend as K

#-- constants
ENV = 'backtrader-v46'

RUN_TIME = 30
THREADS = 8
OPTIMIZERS = 2
THREAD_DELAY = 0.001

GAMMA = 0.99

N_STEP_RETURN = 8
GAMMA_N = GAMMA ** N_STEP_RETURN

EPS_START = 0.4
EPS_STOP  = .15
EPS_STEPS = 75000

MIN_BATCH = 32
LEARNING_RATE = 5e-3

LOSS_V = .5			# v loss coefficient
LOSS_ENTROPY = .01 	# entropy coefficient

#---------
class Brain:
	train_queue = [ [], [], [], [], [] ]	# s, a, r, s', s' terminal mask
	lock_queue = threading.Lock()

	def __init__(self):
		self.session = tf.Session()
		K.set_session(self.session)
		K.manual_variable_initialization(True)

		self.model = self._build_model()
		self.graph = self._build_graph(self.model)

		self.session.run(tf.global_variables_initializer())
		self.default_graph = tf.get_default_graph()

		self.default_graph.finalize()	# avoid modifications

	def _build_model(self):

		l_input = Input( batch_shape=(None, NUM_STATE) )
		l_dense = Dense(16, activation='relu')(l_input)

		out_actions = Dense(NUM_ACTIONS, activation='softmax')(l_dense)
		out_value   = Dense(1, activation='linear')(l_dense)

		model = Model(inputs=[l_input], outputs=[out_actions, out_value])
		model._make_predict_function()	# have to initialize before threading

		return model

	def _build_graph(self, model):
		s_t = tf.placeholder(tf.float32, shape=(None, NUM_STATE))
		a_t = tf.placeholder(tf.float32, shape=(None, NUM_ACTIONS))
		r_t = tf.placeholder(tf.float32, shape=(None, 1)) # not immediate, but discounted n step reward
		
		p, v = model(s_t)

		log_prob = tf.log( tf.reduce_sum(p * a_t, axis=1, keep_dims=True) + 1e-10)
		advantage = r_t - v

		loss_policy = - log_prob * tf.stop_gradient(advantage)									# maximize policy
		loss_value  = LOSS_V * tf.square(advantage)												# minimize value error
		entropy = LOSS_ENTROPY * tf.reduce_sum(p * tf.log(p + 1e-10), axis=1, keep_dims=True)	# maximize entropy (regularization)

		loss_total = tf.reduce_mean(loss_policy + loss_value + entropy)

		optimizer = tf.train.RMSPropOptimizer(LEARNING_RATE, decay=.99)
		minimize = optimizer.minimize(loss_total)

		return s_t, a_t, r_t, minimize

	def optimize(self):
		if len(self.train_queue[0]) < MIN_BATCH:
			time.sleep(0)	# yield
			return

		with self.lock_queue:
			if len(self.train_queue[0]) < MIN_BATCH:	# more thread could have passed without lock
				return 									# we can't yield inside lock

			s, a, r, s_, s_mask = self.train_queue
			self.train_queue = [ [], [], [], [], [] ]

		s = np.vstack(s)
		a = np.vstack(a)
		r = np.vstack(r)
		s_ = np.vstack(s_)
		s_mask = np.vstack(s_mask)

		if len(s) > 5*MIN_BATCH: print("Optimizer alert! Minimizing batch of %d" % len(s))

		v = self.predict_v(s_)
		r = r + GAMMA_N * v * s_mask	# set v to 0 where s_ is terminal state
		
		s_t, a_t, r_t, minimize = self.graph
		self.session.run(minimize, feed_dict={s_t: s, a_t: a, r_t: r})

	def train_push(self, s, a, r, s_):
		with self.lock_queue:
			self.train_queue[0].append(s)
			self.train_queue[1].append(a)
			self.train_queue[2].append(r)

			if s_ is None:
				self.train_queue[3].append(NONE_STATE)
				self.train_queue[4].append(0.)
			else:	
				self.train_queue[3].append(s_)
				self.train_queue[4].append(1.)

	def predict(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)
			return p, v

	def predict_p(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)		
			return p

	def predict_v(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)		
			return v

#---------
frames = 0
class Agent:
	def __init__(self, eps_start, eps_end, eps_steps):
		self.eps_start = eps_start
		self.eps_end   = eps_end
		self.eps_steps = eps_steps

		self.memory = []	# used for n_step return
		self.R = 0.

	def getEpsilon(self):
		if(frames >= self.eps_steps):
			return self.eps_end
		else:
			return self.eps_start + frames * (self.eps_end - self.eps_start) / self.eps_steps	# linearly interpolate

	def act(self, s):
		eps = self.getEpsilon()			
		global frames; frames = frames + 1

		if random.random() < eps:
			return random.randint(0, NUM_ACTIONS-1)

		else:
			s = np.array([s])
			p = brain.predict_p(s)[0]

			# a = np.argmax(p)
			a = np.random.choice(NUM_ACTIONS, p=p)

			return a
	
	def train(self, s, a, r, s_):
		def get_sample(memory, n):
			s, a, _, _  = memory[0]
			_, _, _, s_ = memory[n-1]

			return s, a, self.R, s_

		a_cats = np.zeros(NUM_ACTIONS)	# turn action into one-hot representation
		a_cats[a] = 1 

		self.memory.append( (s, a_cats, r, s_) )

		self.R = ( self.R + r * GAMMA_N ) / GAMMA

		if s_ is None:
			while len(self.memory) > 0:
				n = len(self.memory)
				s, a, r, s_ = get_sample(self.memory, n)
				brain.train_push(s, a, r, s_)

				self.R = ( self.R - self.memory[0][2] ) / GAMMA
				self.memory.pop(0)		

			self.R = 0

		if len(self.memory) >= N_STEP_RETURN:
			s, a, r, s_ = get_sample(self.memory, N_STEP_RETURN)
			brain.train_push(s, a, r, s_)

			self.R = self.R - self.memory[0][2]
			self.memory.pop(0)	
	
	# possible edge case - if an episode ends in <N steps, the computation is incorrect
		
#---------
class Environment(threading.Thread):
	stop_signal = False

	def __init__(self, instancenumber=0, render=False, eps_start=EPS_START, eps_end=EPS_STOP, eps_steps=EPS_STEPS):
		threading.Thread.__init__(self)

		self.render = render

		#-- create btgym environment
		self.env = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
		                  start_weekdays=[0, 1, 2],
		                  episode_len_days=1,
		                  episode_len_hours=23,
		                  episode_len_minutes=55,
		                  start_00=True,
		                  start_cash=100,
		                  broker_commission=0.002,
		                  fixed_stake=10,
		                  drawdown_call=30,
		                  state_shape=(4,30),
		                  state_low=None,
		                  state_high=None,
		                  port=5555 + instancenumber,
		                  verbose=1,)

		self.agent = Agent(eps_start, eps_end, eps_steps)

	def runEpisode(self):
		s = self.env.reset()
		print(s)

		R = 0
		while True:         
			time.sleep(THREAD_DELAY) # yield 

			if self.render: self.env.render()

			a = self.agent.act(s)
			s_, r, done, info = self.env.step(a)

			if done: # terminal state
				s_ = None

			self.agent.train(s, a, r, s_)

			s = s_
			R += r

			if done or self.stop_signal:
				break

		print("Total R:", R)

	def run(self):
		while not self.stop_signal:
			self.runEpisode()

	def stop(self):
		self.stop_signal = True

#---------
class Optimizer(threading.Thread):
	stop_signal = False

	def __init__(self):
		threading.Thread.__init__(self)

	def run(self):
		while not self.stop_signal:
			brain.optimize()

	def stop(self):
		self.stop_signal = True




#-- main
env_test = Environment(render=True, instancenumber=0, eps_start=0., eps_end=0.)
print(env_test.env.observation_space.shape[0])
NUM_STATE = env_test.env.observation_space.shape[0]
NUM_ACTIONS = env_test.env.action_space.n
NONE_STATE = np.zeros(NUM_STATE)

brain = Brain()	# brain is global in A3C

envs = [Environment(instancenumber=i) for i in range(THREADS)]
opts = [Optimizer() for i in range(OPTIMIZERS)]

for o in opts:
	o.start()

for e in envs:
	e.start()

time.sleep(RUN_TIME)

for e in envs:
	e.stop()
for e in envs:
	e.join()

for o in opts:
	o.stop()
for o in opts:
	o.join()

print("Training finished")
env_test.run()

Very great project, thank you for your pay.
In the example of unreal_stacked_lstm_strat_4_11, is it faster to use GPU computing than CPU?
The effect of using a GPU does not seem to be good.

Hi, thanks for another new great gym environment!

Its not actual issue, more like question:
I am curious about actual RL trained strategies examples (not necessary profitable), because I didn't find any and OpenAI didn't pay much attention to financial gyms. It will be much easier to learn on existed examples how tune architecture or parameters. I would be very grateful if you point me out useful links if you know some. Thanks

Hi,

great work!!
One question: I was trying to test a long only agent but it seems setting the portfolio_action, i.e.:
MyCerebro.addstrategy(MyStrategy, drawdown_call=10, skip_frame=1,state_shape=(10,4),portfolio_actions=('hold', 'buy','close'))
does not work and the agent continues with short selling.

Could you provide an example explicitly disallowing short selling?

Thanks!

Hi, nice work with btgym, I was working in a similar project (https://github.com/AdrianP-/gym_trading) and I wanted to integrate RL with a zipline/pyalgotrader/etc, but you already did it! Please send me a email :)

However I tried your examples and my kernel always die in MyEnvironment = gym.make('backtrader-v46'). Commenting this line everything works well :D. Why is necessary?

vvv Scroll down for latest updates vvv

• 6.02.18: Pushed common update to all a3c agents architectures:
  • all dense layers are now Noisy-Net ones,
    see: Noisy Networks for Exploration paper by Fortunato at al.;
    • note that entropy regularization is still here, lowered to ~0.01 to ensure proper exploration;
      update your entropy_beta custom settings, if done any;
  • policy output distribution is 'centered' using layer normalisation technique;
  • added per-channel weights scaling to obs state before passing to tanh, see #35 and https://github.com/Kismuz/btgym/blob/master/examples/state_signal_scaling.ipynb

All of the above results in about 2x training speedup in terms of train iterations: stacked LSTM agents takes ~20M steps to converge on 6 month real data set (was: 40).

I found that the default action space is ['hold', 'close', 'buy', 'sell']. so how can I invest n assets with different portfolio weights?

Hi again,

I'm still having problems running the basic examples. I noticed you're running python 3 judging on the leftover pyc files. I'm trying with 2.7.11, latest dependencies. It could be a Python 3 vs 2.7 thing but I can't see why, to me it looks like you're not including backtrader when referencing it (but I can see it is included). Somehow it works for you but not me. Example error below:

from btgym import BTgymEnv
​

Simpliest trading environment,

using year-long dataset of one minute bars for EUR/USD currency pair:

​

MyEnvironment = BTgymEnv(filename='./data/DAT_ASCII_EURUSD_M1_2016.csv',)
​

Print parameters and their default values:

​
for params_name, params_dict in MyEnvironment.params.items():
print('\nParameters [{}] subset: '.format(params_name))
for key, value in params_dict.items():
print('{} : {}'.format(key,value))

AttributeError Traceback (most recent call last)
in ()
5
6
----> 7 MyEnvironment = BTgymEnv(filename='./data/DAT_ASCII_EURUSD_M1_2016.csv',)
8
9 # Print parameters and their default values:

btgym/envs/backtrader.pyc in init(self, **kwargs)
214 # if no bt.Cerebro() custom subclass has been passed,
215 # get base class Cerebro(), using kwargs on top of defaults:
--> 216 self.engine = bt.Cerebro()
217 self.params['engine'].update(self.kwargs['engine'])
218 msg = 'Base Cerebro engine used.'

AttributeError: 'module' object has no attribute 'Cerebro'

I then try my basic example from my previous issue I get the following error:

Traceback (most recent call last):
File "test.py", line 27, in
verbose=1)
File "btgym/envs/backtrader.py", line 261, in init
if self.engine.strats[0][0][2]['state_low'] is None or
KeyError: 'state_low'

Any clues? I'm pulling my hair out as I can initialise bt separately in Jupyter and I get the same errors regardless of jupyter or command line.

If you think in algo-trader terms, the normal is taking an action every time that you get a new candle, so the Agent should do the same. But btgym isn't prepared for state_shape=(1, 4)
There is a reason for that?

Running environment:

OS: Ubuntu 16.0.4
tensorflow v1.7.0
Nvidia Driver Version: 390.48
Cuda : v9.1.85

Files or part of package has been run:

guided_a3c.py

Actual behaviour:

Traceback (most recent call last):
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 671, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 4 but is rank 3 for 'global/MaxPool' (op: 'MaxPool') with input shapes: [1,28,1].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/aac.py", line 405, in init
self.network = self._make_policy('global')
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/aac.py", line 807, in _make_policy
network = self.policy_class(**self.policy_kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/research/gps/policy.py", line 31, in init
**kwargs
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/policy/stacked_lstm.py", line 460, in init
super(AacStackedRL2Policy, self).init(**kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/policy/stacked_lstm.py", line 370, in init
pixel_change_2d_estimator(ob_space['external'], **kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/nn/networks.py", line 249, in pixel_change_2d_estimator
'SAME'
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1769, in max_pool
name=name)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1605, in _max_pool
data_format=data_format, name=name)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2508, in create_op
set_shapes_for_outputs(ret)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1873, in set_shapes_for_outputs
shapes = shape_func(op)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1823, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
debug_python_shape_fn, require_shape_fn)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Shape must be rank 4 but is rank 3 for 'global/MaxPool' (op: 'MaxPool') with input shapes: [1,28,1].

I have trained the model. Can I use the trained agent to generate Actions in the Running agent section of setting_up_environment_basic?

Andrew, first thank you for your work! In the last commit initialization env crashes with an error. What could be the problem? Thanks in advance.

MyCerebro = bt.Cerebro()

MyCerebro.addstrategy(
    DevStrat_4_6,
    drawdown_call=5, # max % to loose, in percent of initial cash
    target_call=10,  # max % to win, same
    skip_frame=10,
)

MyDataset = BTgymDataset(
    #filename='.data/DAT_ASCII_EURUSD_M1_201703.csv',
    #filename='./data/DAT_ASCII_EURUSD_M1_201704.csv',
    filename='./data/test_sine_1min_period256_delta0002.csv',
    start_weekdays={0, 1, 2, 3},
    episode_duration={'days': 0, 'hours': 23, 'minutes': 55},
    start_00=False,
    time_gap={'hours': 6},
)

kwargs=dict(
         dataset=MyDataset,
         engine=MyCerebro,
         render_modes=['episode', 'human','external'],
         render_state_as_image=True,
         render_ylabel='OHL_diff.',
         render_size_episode=(12,8),
         render_size_human=(9, 4),
         render_size_state=(11, 3),
         render_dpi=75,
         port=5000,
         data_port=4999,
         connect_timeout=60,
         verbose=0,  # better be 0
     )
env = BTgymEnv(**kwargs)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-2abf37cd704a> in <module>()
----> 1 env = BTgymEnv(**kwargs)

/var/www/btgym/btgym/envs/backtrader.py in __init__(self, *args, **kwargs)
    334 
    335         # Set observation space shape from engine/strategy parameters:
--> 336         self.observation_space = DictSpace(self.params['strategy']['state_shape'])
    337 
    338         self.log.debug('Obs. shape: {}'.format(self.observation_space.spaces))

/var/www/btgym/btgym/spaces.py in __init__(self, spaces)
     37         """
     38         super(DictSpace, self).__init__(spaces)
---> 39         self.shape = self._get_shape()
     40 
     41     def _get_shape(self):

/var/www/btgym/btgym/spaces.py in _get_shape(self)
     40 
     41     def _get_shape(self):
---> 42         return OrderedDict([(k, space.shape) for k, space in self.spaces.items()])
     43 
     44 

/var/www/btgym/btgym/spaces.py in <listcomp>(.0)
     40 
     41     def _get_shape(self):
---> 42         return OrderedDict([(k, space.shape) for k, space in self.spaces.items()])
     43 
     44 

AttributeError: 'Dict' object has no attribute 'shape'

Dear Sir

Could you please help me fix this problem ?

I tried running a3c_sinewave but when i run the last cell (launcher.run())

I found this problem.
[2017-09-04 19:24:45,266] Data file <../examples/data/test_sine_1min_period256_delta0002.csv> not specified / not found.
Process BTgymDataFeedServer-2:1:
Traceback (most recent call last):
File "/home/balldekdee/btgym/btgym/datafeed.py", line 127, in read_csv
assert filename and os.path.isfile(filename)
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/balldekdee/btgym/btgym/datafeed.py", line 142, in read_csv
assert 'episode_dataset' in self.filename
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/balldekdee/btgym/btgym/dataserver.py", line 73, in run
self.dataset.read_csv()
File "/home/balldekdee/btgym/btgym/datafeed.py", line 149, in read_csv
raise FileNotFoundError(msg)
FileNotFoundError: Data file <../examples/data/test_sine_1min_period256_delta0002.csv> not specified / not found.
[2017-09-04 19:24:50,508] Press Ctrl-C to stop training and close launcher.

When using the parameter state_shape to create BTgymEnv, it will throw AttributeError: 'tuple' object has no attribute 'keys'

Running environment:

python 3, macosx

from btgym import BTgymEnv

MyEnvironment = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
episode_duration={'days': 2, 'hours': 23, 'minutes': 55},
drawdown_call=50,
state_shape=(4,20),
port=5555,
verbose=1,
)

Files or part of package has been run:

backtrader.py

Actual behaviour:

[2018-04-19 01:04:56.697574] INFO: BTgymAPIshell_0: ...done.
Traceback (most recent call last):
[2018-04-19 01:04:56.698795] INFO: BTgymAPIshell_0: Base Cerebro class used. Base Strategy class used.
File "/btgym/tests/quickstart.py", line 8, in
verbose=1,
File "/btgym/btgym/envs/backtrader.py", line 353, in init
if 'raw_state' in self.params['strategy']['state_shape'].keys():
AttributeError: 'tuple' object has no attribute 'keys'

I have a problem to run the code, after the first epoch, I call the function env.reset() to start a new one but I have the following error :

WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
WARN: Environment '<class 'btgym.envs.backtrader.BTgymEnv'>' has deprecated methods. Compatibility code invoked.
learning-rules.py:320: RuntimeWarning: overflow encountered in multiply
self.states = self.states+ learning_rateestimated_valuetraces
learning-rules.py:320: RuntimeWarning: invalid value encountered in multiply
self.states = self.states+ learning_rateestimated_valuetraces
learning-rules.py:320: RuntimeWarning: invalid value encountered in add
self.states = self.states+ learning_rateestimated_valuetraces
Episode 1 finished
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/lines.py:44: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(style) and is_hashable(style):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/style/core.py:92: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if cbook.is_string_like(style) or hasattr(style, 'keys'):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/style/core.py:99: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if not cbook.is_string_like(style):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/font_manager.py:971: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(family):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/font_manager.py:697: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(family):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/text.py:218: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
elif is_string_like(fontproperties):
Process BTgymServer-2:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 453, in run
episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
self._runnext(runstrats)
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
strat._next()
File "/usr/local/lib/python3.5/dist-packages/backtrader/strategy.py", line 328, in _next
self._next_analyzers(minperstatus)
File "/usr/local/lib/python3.5/dist-packages/backtrader/strategy.py", line 362, in _next_analyzers
analyzer._next()
File "/usr/local/lib/python3.5/dist-packages/backtrader/analyzer.py", line 188, in _next
self.next()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 158, in next
self.early_stop()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 80, in early_stop
self.render.render(self.render_at_stop, step_to_render=self.step_to_render, send_img=False)
File "/home/nicolas/Documents/trading/btgym/btgym/rendering/renderer.py", line 284, in render
xlabel=self.render_xlabel,
File "/home/nicolas/Documents/trading/btgym/btgym/rendering/renderer.py", line 312, in draw_plot
self.plt.title(title)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 1465, in title
return gca().set_title(s, *args, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 950, in gca
return gcf().gca(**kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/figure.py", line 1369, in gca
return self.add_subplot(1, 1, 1, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/figure.py", line 1021, in add_subplot
a = subplot_class_factory(projection_class)(self, *args, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 73, in init
self._axes_class.init(self, fig, self.figbox, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 529, in init
self._init_axis()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 622, in _init_axis
self.xaxis = maxis.XAxis(self)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 676, in init
self.cla()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 760, in cla
self.reset_ticks()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 771, in reset_ticks
cbook.popall(self.majorTicks)
AttributeError: module 'matplotlib.cbook' has no attribute 'popall'
Traceback (most recent call last):
File "learning-rules.py", line 335, in
q.learning(env)
File "learning-rules.py", line 304, in learning
current_state = env.reset()
File "/home/nicolas/Documents/trading/btgym/btgym/envs/backtrader.py", line 595, in _reset
if self._force_control_mode():
File "/home/nicolas/Documents/trading/btgym/btgym/envs/backtrader.py", line 509, in _force_control_mode
self.server_response = self.socket.recv_pyobj()
File "/home/nicolas/.local/lib/python3.5/site-packages/zmq/sugar/socket.py", line 491, in recv_pyobj
msg = self.recv(flags)
File "zmq/backend/cython/socket.pyx", line 693, in zmq.backend.cython.socket.Socket.recv
File "zmq/backend/cython/socket.pyx", line 727, in zmq.backend.cython.socket.Socket.recv
File "zmq/backend/cython/socket.pyx", line 150, in zmq.backend.cython.socket._recv_copy
File "zmq/backend/cython/socket.pyx", line 145, in zmq.backend.cython.socket._recv_copy
File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable

Thanks for great work :)

I have an issue while running examples - a3c_random_on_synth_or_real_data ...

I got several <INFO:tensorflow:Error reported to Coordinator: <class 'queue.Empty'>> messages and then stopped.

Is there anyway I can fix it ?? Thank you so much.
Kim.

[2018-01-11 20:50:20,439] Error reported to Coordinator: <class 'queue.Empty'>,
Process Worker-6:
Traceback (most recent call last):
File "/home/joowonkim/anaconda3/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/worker.py", line 241, in run
trainer.process(sess)
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 747, in process
data = self.get_data()
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 594, in get_data
data_streams = [get_it() for get_it in self.data_getter]
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 594, in
data_streams = [get_it() for get_it in self.data_getter]
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/rollout.py", line 33, in pull_rollout_from_queue
return queue.get(timeout=600.0)
File "/home/joowonkim/anaconda3/lib/python3.5/queue.py", line 172, in get
raise Empty
queue.Empty

INFO:tensorflow:global/global_step/sec: 0

[2018-01-11 20:51:38,860] global/global_step/sec: 0

INFO:tensorflow:Error reported to Coordinator: <class 'queue.Empty'>,

[2018-01-11 20:51:48,678] Error reported to Coordinator: <class 'queue.Empty'>,

and stopped

Hi @Kismuz,

Great project! I would like to know if its possible to use the normal btfeeds for backtesting and live feeds within the gym environment?

Is it possible to save and load trained models?

Also is it possible to modify the order size on the fly eg.

            self.buyvalue = round(((self.broker.getvalue() / self.data0.close[0]) * 0.9), 1)
            self.order = self.buy(self.data,
                                  exectype=bt.Order.Limit, 
                                  size=self.buyvalue,
                                  price=self.data0.close[0],
                                  valid=datetime.now() + timedelta(minutes=10))

Thank you in advance,

Kind regards

Is it possible to do Testing on Interactive Brokers (IB) as is ? Just to test with actual trading environment without using actual funds. Backtrader uses IB or OANDA for trading and testing

hi Andrew,

Can this framework support use different stocks with the same date time range as input? can you give me an example?

I mean the input for the framework is:

input_file1:
20170901 133000;15345;15355;15340;15335;868
20170901 133100;15335;15340;15330;15330;346
20170901 133200;15330;15330;15305;15300;3088
20170901 133300;15305;15320;15320;15300;1290
20170901 133400;15315;15320;15305;15305;1154
20170901 133500;15310;15315;15315;15300;2882
20170901 133600;15310;15315;15315;15305;434
20170901 133700;15310;15315;15310;15305;506
20170901 133800;15310;15315;15310;15305;906
20170901 133900;15315;15315;15315;15310;308
20170901 134000;15310;15315;15315;15310;180
20170901 134100;15315;15315;15305;15305;418

input_file2:
20170901 133000;4003;4006;4005;4003;2816
20170901 133100;4005;4006;4004;4003;1500
20170901 133200;4004;4004;4001;4000;3306
20170901 133300;4000;4002;4002;3999;2476
20170901 133400;4001;4002;4001;4001;1036
20170901 133500;4001;4001;4000;4000;1758
20170901 133600;4000;4005;4004;3999;5274
20170901 133700;4004;4005;4005;4003;1322
20170901 133800;4004;4005;4005;4004;802
20170901 133900;4004;4006;4006;4004;3030
20170901 134000;4006;4007;4006;4005;2144
20170901 134100;4006;4007;4006;4005;1068
20170901 134200;4006;4006;4005;4004;354
20170901 134300;4005;4005;4004;4004;690
20170901 134400;4004;4005;4004;4004;594
20170901 134500;4005;4006;4005;4004;1904
20170901 134600;4005;4007;4006;4005;1962
20170901 134700;4006;4008;4008;4006;2412
20170901 134800;4008;4010;4009;4008;2226
20170901 134900;4010;4010;4009;4009;1240
20170901 135000;4009;4009;4008;4008;334
20170901 135100;4008;4008;4007;4007;750

Thank you!

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Ubuntu 16.10 LTS
Python 3.5/3.6

Files or part of package has been run:

setup.py

Expected behaviour:

Everything is expected to work as normal.

Actual behaviour:

After running python3 setup.py develop --user, pip3 breaks with import errors.

Seems to be related to this one pypa/setuptools#885

Steps to reproduce:

Hi, I want to thank you for all that you've been doing for supporting backtrader with a custom OpenAI Gym Environment, it really has helped me a lot.

I'm having difficulties adapting the environment to a new dataset, which is for Bitcoin-USD currency.

MyEnvironment = BTgymEnv(filename='btc_usd.csv')

Here I have added the first 10 lines of my btc_usd.csv, for which I've maintained the same formatting as found in your example csv. I've added volume values to each row as well:

20170911 194700;4209.48;4209.98;4209.48;4209.98;3.0283
20170911 194600;4204.65;4209.49;4204.64;4204.64;1.4119397600000003
20170911 194500;4204.64;4204.65;4204.64;4204.65;7.74892999
20170911 194400;4215.01;4215.01;4203.04;4204.64;11.687842009999995
20170911 194300;4215;4215.01;4215;4215.01;1.6720000000000002
20170911 194200;4215.01;4215.01;4215.01;4215.01;6.270305769999999
20170911 194100;4215.01;4215.01;4215.01;4215.01;0.25924983
20170911 194000;4215.01;4215.01;4215.01;4215.01;1.8750531499999998
20170911 193900;4215.01;4215.01;4215.01;4215.01;0.0867
20170911 193800;4215.21;4215.21;4215;4215.01;5.09584227

The error I'm getting is Pandas related:

Traceback (most recent call last):
  File "/Users/yazkhoury/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/Users/yazkhoury/Desktop/ml-trading/btgym/btgym/dataserver.py", line 89, in run
    episode_dataset = self.dataset.sample_random()
  File "/Users/yazkhoury/Desktop/ml-trading/btgym/btgym/datafeed.py", line 253, in sample_random
    first_row = self.data.index.get_loc(adj_timedate, method='nearest')
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/tseries/index.py", line 1411, in get_loc
    return Index.get_loc(self, key, method, tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2138, in get_loc
    indexer = self.get_indexer([key], method=method, tolerance=tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2262, in get_indexer
    tolerance=tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2271, in get_indexer
    raise InvalidIndexError('Reindexing only valid with uniquely'
pandas.indexes.base.InvalidIndexError: Reindexing only valid with uniquely valued Index objects
[2017-09-14 11:00:03,575] Data_server unreachable with status: <receive_failed>.
Data_server unreachable with status: <receive_failed>.

I've attached a WeTransfer url to the CSV file here: https://we.tl/jWPBGGwyzT

I'm just curious as to what the problem could be in Pandas end? I verified that the indexes aren't duplicated for the date-time, and I've tested out the environment on one of your example datasets, and it works perfectly.

I think it might be a formatting issue in my CSV but I'm really not sure. I've saved my CSV file as:
datetime, open, high, low, close, volume for each row.