Asynchronous Methods for Deep Reinforcement Learning

Playing the games simultaneously is actually innovative in how it let's them learn faster. If you play one game at a time consecutive moves are highly correlated which makes it difficult to make learning updates after each move. If you're playing 1250 parallel games that are starting and restarting so that they're at all at different stages of play, you can do an update after every move in every game and make better progress faster. This is known as Asynchronous Advantage Actor-Critic (A3C) and is a highly cited method discovered by the DeepMind team [1]. [1] https://arxiv.org/abs/1602.01783

> we were able to solve one of the hardest MuJoCo tasks (a 3D humanoid) using 1,440 CPUs across 80 machines in only 10 minutes. As a comparison, in a typical setting 32 A3C workers on one machine would solve this task in about 10 hours. So using 80 times more machines makes you 60 times faster (assuming those are the same machines) "while performing better on 23 games tested, and worse on 28"[0]? [0] The paper for this blog post https://arxiv.org/pdf/1703.03864.pdf Asynchronous advantage actor critic (A3C) https://arxiv.org/pdf/1602.01783.pdf

What is state of the art in reinforcement learning right now? https://arxiv.org/abs/1602.01783 Is there a way to deal with "sparse" training data (state, action, reward) triples -- sparse in "state"?

There was a paper a few months back in deep reinforcement learning that got record setting RL results using only a CPU [0]. Previously, these algorithms would play fewer games and run a gpu over and over on the few games they had played. By using a CPU you can generate more samples that are up to date with your learning algorithm. It sounds obvious in hindsight, but you can't exactly run 60 atari sims on a gpu. [0] - https://arxiv.org/abs/1602.01783

Couple key points: This project[0] is going to keep developing. It has Deep Q Learning now, A3C[1] is working and more features are being added. Just as important: RL4J and DL4J run on a scientific computing framework called ND4J[2] that integrates with Spark and trains on multiple GPUs.[3][4] It's basically porting RL to the production stack of large organizations that work with the JVM and need to scale. The key thing to remember is that RL combines with other algorithms, like deep convolutional nets or Monte Carlo Search Trees. DL4J has the ConvNets already.[5] [0] https://github.com/deeplearning4j/rl4j [1] https://arxiv.org/abs/1602.01783 [2] http://nd4j.org/ [3] http://deeplearning4j.org/spark [4] http://deeplearning4j.org/gpu [5] http://deeplearning4j.org/convolutionalnets.html

Highlights: 1) No GPUs necessary, gives similar/better results in similar time (see http://arxiv.org/pdf/1602.01783v1.pdf) 2) 1-step Q (according to paper, maybe could do better with actor-critic + 1 LSTM layer) 3) Keras is really only used for a tiny fraction of the code. Which means that there is a lot of boilerplate that could be rolled up into a shared library still.