End-to-End Training of Deep Visuomotor Policies

made a ~3d KGS policy network into a ~5d KGS bot, and used to generate a training sample for the value net. If we had enough recorded human games to train the value net directly, that'd be an unnecessary step anyhow. And you could create such a training set w/o reinforcement learning since there are pure monte carlo bots stronger than 5d KGS - but that'd be far more computationally expensive. But its still not really true that there aren't obvious applications of deep reinforcement learning - indeed robotics is one promising application, and that seems rather relevant. this paper initially demonstrated an impressive improvement in manipulative tasks, and you can prob follow its numerous citations for newer stuff: http://arxiv.org/abs/1504.00702 I do agree that this exact architecture in AlphaGo prob doesn't have applications beyond just teaching us how to play go better; it seems too specialized. I believe they mean it in just the vaguest possible sense; that the kind of deep algorithms demonstrating incredible performance in AlphaGo have diverse applications; but this should not come as a surprise to anyone even loosely following what people have done with deep learning in the past couple of years anyhow.

trajectories through motor-torque space to produce a desired goal are intriguingly like planning. Mikolov's[3] Word2vec & Radford's Image Vectors[4] produce semantic spaces where vector math is akin to reasoning. Higher level control could be achieved through vector algebras deriving actions and things directly from the robots internal vector space of motor-torques and raw-pixels Thus internal vectors are like thoughts[5] which can interpret, reason, plan, and control. Karpathy shows internal vectors can bridge modalities from words to pictures[6]. If the higher level control vectors of actions and things could translate to the modality of language as verbs and nouns, perhaps the robot could discuss its plans, and receive orders. [1] - Thelen's Infant Development http://www.ncbi.nlm.nih.gov/pubmed/17107442 [2] - Levine, Finn, Darrel, Abeel http://arxiv.org/abs/1504.00702 & [2b] - Distributed Grasp Learning http://googleresearch.blogspot.co.uk/2016/03/deep-learning-f... [3] - Mikolov, Chen, Corrado, Dean http://arxiv.org/abs/1301.3781 [4] - Radford, Metz, Chintala https://github.com/Newmu/dcgan_code#arithmetic-on-faces [5] - Hinton https://youtu.be/IcOMKXAw5VA?t=32m55s [6] - Karpathy https://youtu.be/ZkY7fAoaNcg?t=38m31s

can be used for path planning [4] (with tire slippage, etc.). So his system will make errors in emergency situations. Especially since the neural net has never experienced most emergencies and could not learn the appropriate reactions. And guess what, emergency situations are the hard part. Driving on a freeway with visible lane markings is easy. German research projects autonomously drove on the Autobahn since the 80s [5]. Neural networks were used for the task since about the same time [6]. [1] http://www.instructables.com/id/Hack-your-vehicle-CAN-BUS-wi... [2] http://arxiv.org/abs/1509.02971 [3] http://arxiv.org/abs/1504.00702 [4] http://www.rem2030.de/rem2030-wAssets/docs/downloads/07_Konf... [5] https://en.wikipedia.org/wiki/Eureka_Prometheus_Project [6] http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&c...

to do. And unlikely a classification system where the output space is relatively small (ImageNet has 1000 classes), an RL agent is searching over an exponentially large space of possible trajectories. Which means that without some additional source of supervision you can spend a long time wandering in the wilderness with no idea of whether what you're doing is reasonable or how to get any reward at all. And when you do get some reward, you have no idea which of the possibly hundreds or thousands of actions you took deserves the credit. A lot of recent RL research is about finding additional sources of supervision, such as training an agent to mimic the policy of an "expert" (e.g., a search algorithm that runs in a simulator to find an optimal solution, but which requires too much computation to actually apply directly at test time, as in http://arxiv.org/abs/1504.00702), or coming up with proxy objectives like "empowerment" or "curiosity" (which you can define in terms of information theoretic quantities, such as mutual information between the agents' action sequence and future state, e.g. http://arxiv.org/abs/1509.08731) to supplement the actual reward signal. This latter path, the notion of "intrinsic reward" that paulsutter alluded to, is in some sense the merger of RL with unsupervised learning, and a lot of the power is that you're getting new reward signals constantly, not just when you finally manage to achieve some arbitrarily difficult task.

contactrich manipulation skills with guided policy search. In International Conference on Robotics and Automation(ICRA), 2015. I haven't read that paper yet. Interestingly, they initialise the visual learning model using the ImageNet images. Was it 3 years ago that was considered a pretty much intractable problem, and now the fact a CNN can work on it well enough to be useful isn't even worth a complete sentence. [1] http://arxiv.org/pdf/1504.00702v1.pdf

have been ? Edinburgh had both symbolic A.I. planning, a working vision system & pragmatic bottom up machine learning until '73 & Lighthill. Edinburgh's was a unique approach - with bigger budgets the American labs favoured simulation (until 1985 when Rod Brooks showed it was a dead end) and without BOXES, Freddy became RAPT, symbolic, programmed GOFAI & no learning. RAPT failed because top-down reasoning was a computational quagmire in the real world & it was too fragile to the slightest decalibration. Edinburgh's hybrid GOFAI & behavior-based approaches got round calibration issues with SOMASS, Eddie & Push-Slide. But programmed solutions lacked the crucial element of learning - Michie's BOXES. Concurrently, Michie, exiled to Glasgow & private industry, found great success helping the Japanese revolutionise the assembly line & in the US with automated factory control & satellite piloting for NASA. All reinforcement learnt using BOXES but he & AI were pariah in the UK as he laments in 1987: [Michie from 18:24] http://www.bbc.co.uk/iplayer/episode/p0306rt1/micro-live-ser... Michie's nascent hybrid of raw sensor machine learning & GOFAI embodied in robots is just being rediscovered in Abeel & Levines work today. http://arxiv.org/abs/1504.00702 An AI winter is a terrible thing. Progress lost decades. Tragically the 2002 fire may have destroyed Michie's undigitised archive.