alex-petrenko / landmark-exploration Goto Github PK

Might be an interesting environment to try
https://blogs.unity3d.com/2019/01/28/obstacle-tower-challenge-test-the-limits-of-intelligence-systems/

Based on distance metric we can provide both sparse rewards (e.g. only +1 for discovering new landmarks) and dense reward (e.g. give a positive reward for getting further from the known landmarks).

Intuitively, sparse rewards should be more reliable, because with dense reward there might exist unexpected local maxima. Although with dense rewards we can achieve better sample efficiency.

The idea of state-space map is pretty general, and we can try an experiment with sparse-reward robotic manipulation (e.g. pushing an object to a target, stacking, etc.)

Need wrappers and default algorithm parameters

For the generalization study, we need elaborate randomly generated mazes, we can use VizDoom level generator for this.

This should be pretty straightforward. We already have an implementation of ICM with A2C, need to use it with PPO algorithm. RND is the same but without training of the inverse dynamics model.

Currently, neighbors are stored as plain images. Instead, it should be better to store them as feature vectors.

In DMLab we can easily query ground truth coordinates of the agent with respect to the map.
Would be very cool to adjust the graph visualization to incorporate this information.

Space discretization can be implemented in two different ways:

1. (current implementation) train a binary classifier, 0 - observations are close, 1 - observations are far apart
2. learn an embedding space and use the distance in embedding space to determine the reachability

(2) has the potential to be more generalizable, and I've never seen people use it this way.

We randomly generate a big number of 3D mazes, split them into train, test and validation sets. Then the policy is only trained on a train set.

We can open-source this dataset, which should be a good contribution.

The agent should be able to robustly navigate between any pair of landmarks connected by an edge in the graph.
Possibilities are:

1. Collect past trajectories from exploration policy that initially discovered the landmarks and train locomotion using behaviour cloning
2. Train locomotion policy with RL: reward +1 is given when target is reached (based on distance metric).

This mostly applies to training on a single environment. We need to both learn an exploration policy (that would know how to expand the graph), but we also need to make progress so we should not reset the graph too often.

Currently implemented with the dynamic RNN, but we can also do deep sets and potentially other architectures.

When the exploration is stuck we want to look at the graph and select the next target for exploration. A basic policy can be to just randomly sample one, although might not be efficient for large environments, because we always want to be on the "frontier" of exploration.

One idea is to use the value estimate of the exploration policy as the "potential" of the landmark for further exploration. We can also use UCB or Thompson sampling to make sure we explore all landmarks, even those that don't seem promising now.

This should be TMAX but without:

1. Getting rewards for finding edges in the graph
2. Using locomotion policy
3. Using neighborhood encoder

I think we should just have a flag that would disable all of the above.

The baseline methods were designed mostly to explore a single sparse-rewards environment rather than learn a general exploration policy.
We could generate a number of particularly challenging mazes (or use this Unity environment) to see how our method compares to baselines when trained and evaluated on a single environment.

Now it is marked as "skipped"

The first version will likely be based on just randomly picking a new landmark to explore.

The current assumption is that information about neighbors can allow us to do more "directed" exploration: policy can decide where to go based on what it had already seen in the immediate vicinity. This, however, needs to be verified.

Train the exact same policy, but without the heuristic of returning to the next potentially interesting target.

Now the edges are undirected, which does not make sense in Montezuma, because many transitions in this game are irreversible (e.g. result in death).