How to randomly collect actions from the environment in train_step about tianshou HOT 6 OPEN

Do you mean adding exploration noise during training rollouts? It's a bit tricky and one of the heftier technical debts of tianshou. If that's what you mean, I'll write a short explanation of the current status here

from tianshou.

Thank you for your prompt and professional reply!
According to my understanding, tianshou.trainer.BaseTrainer.train_step() implements the use of train_collector.collect() to collect training data from the environment ,
The usage is as shown in the code below（tianshou/trainer/base.py line 436）：

result = self.train_collector.collect(

          n_step=self.step_per_collect,

          n_episode=self.episode_per_collect,

      )

in collect() There is a piece of code as follows(tianshou/data/collector.py line 285)

if random:

              try:

                  act_sample = [self._action_space[i].sample() for i in ready_env_ids]

              except TypeError:  # envpool's action space is not for per-env

                  act_sample = [self._action_space.sample() for _ in ready_env_ids]

              act_sample = self.policy.map_action_inverse(act_sample)  # type: ignore

              self.data.update(act=act_sample)

My question is how to pass random parameters to collect() function in train_step when using BaseTrainer Class
The method of using trainer in the outermost layer is as follows I can't find how to pass in a random tag

result = OffpolicyTrainer(
      policy=policy,
      train_collector=train_collector,
      test_collector=test_collector,
      max_epoch=args.epoch,
      step_per_epoch=args.step_per_epoch,
      step_per_collect=args.step_per_collect,
      episode_per_test=args.test_num,
      batch_size=args.batch_size,
      update_per_step=args.update_per_step,
      stop_fn=stop_fn,
      train_fn=train_fn,
      test_fn=test_fn,
      save_best_fn=save_best_fn,
      logger=logger,
  ).run()

from tianshou.

I see, I'll check it and will write an explanation tomorrow in the morning. Thanks for the good question, we'll add some documentation (or maybe even a small refactoring, if possible) to make that clearer

from tianshou.

Thanks!

from tianshou.

May I ask what your intended use-case is? If it is collecting warm-up steps for OffPolicy algorithms, these are currently collected before the Collector is handed to the Trainer. Have a look at the examples for algorithms like TD3 or SAC.

from tianshou.

Sorry, the last days were very full, only got around to it now.

1. When instantiating BaseTrainer directly, you will have to create a Collector and pass it as train_collector. Then you can pass random=True when instantiating the corresponding Colllector instance.
2. Currently the train_step interface doesn't allow any additional controls. We might adjust it in the near future. If you want to hack your way through it, you could manually adjust the wrapped train_collector. So you could do

my_trainer.train_collector.random = True
# Do something that you want
my_trainer.train_step()
my_trainer.train_collector.random = False

While it's not pretty, it will work. If you want to use the high-level interfaces, such hacking would be more difficult, but probably still possible

To better understand how we can adjust the interfaces for improving user experience, it would be great to understand your use case, as @maxhuettenrauch has pointed out.

Hope this answer helps you :) @DLoveS1314

from tianshou.