Skip to content
The methodOpen data
Back to the collection

End to End Learning for Self-Driving Cars

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski

Show all 13 authors

Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba

2016 · arXiv

Machine learningComputer vision
Read the paper
HN score
2.5
Mentions
8
Readers
5
Months
7
What’s in a score?

From the abstract

We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

8 selected excerpts · 8 total mentions

cr0sh
Strong recommendation · rule
I can understand why it probably isn't on the list yet (not as many citations, since it is fairly new) - but NVidia's "End to End Learning for Self-Driving Cars" needs to be mentioned, I think: https://arxiv.org/abs/1604.07316 https://images.nvidia.com/content/tegra/automotive/images/20... I implemented a slight variation on this CNN using Keras and TensorFlow for the third project in term 1 of Udacity's Self-Driving Car Engineer nanodegree course (not special in that regard - it was a commonly used implementation, as it works). Give it a shot yourself - take this paper, install TensorFlow, Keras, and Python, download a copy of Udacity's Unity3D car simulator (it was…
Read the full comment
recently released on GitHub) - and have a shot at it! Note: For training purposes, I highly recommend building a training/validation set using a steering wheel controller, and you'll want a labeled set of about 40K samples (though I have heard you can get by with much fewer, even unaugmented - my sample set actually used augmentation of about 8k real samples to boost it up to around 40k). You'll also want to use GPU and/or a generator or some other batch processing for training (otherwise, you'll run out of memory post-haste).
Original comment The most cited deep learning papers
edshiro
Recommendation · rule
NVidia published a very interesting paper on this technique called "End-To-End Learning For Self-Driving Cars": https://arxiv.org/abs/1604.07316 . It's an enjoyable read. The car learns to steer itself on an empty road. It's a good experiment to witness the power of deep learning and neural nets. For autonomous vehicles though, you need much more than that (e.g. sensor fusion, obstacle detection, localization, behaviour prediction, trajectory prediction, path planning, motion control, etc.).
Original comment A self-driving toy car using end-to-end learning
Isinlor
Neutral mention · rule
I'm curious why author decided to completely disregard reinforcement learning. "Note that neural nets are neither. There has been a relatively small amount of non-mainstream work of getting neural nets to control very simple robots, mostly in simulation only." This is just plain false. Playing Arcade Games involves controlling an agent and there is a lot of widely published, very mainstream work from DeepMind, OpenAI and many other less known research groups. The big part of hype is about using deep neural networks in tandem with reinforcement learning. It's a very reactive system. In complex environment a work from Nvidia is notable "End to End Learning for Self-Driving…
Read the full comment
Cars": https://arxiv.org/abs/1604.07316 Also, Alpha Zero is very clever combination of reinforcement learning, neural networks and symbolic reasoning (tree search). Alpha Zero is not only reactive, but also deliberative. Interestingly Alpha Zero, in very limited domain, managed to surpass not only unaided humans, but many years of programmers effort based on purely symbolic methods by beating Stockfish in Chess. The author is no doubt a world class expert in the Good Old AI, but to my surprise he seems to be out of loops with more recent advancements. Or am I missing something / misunderstood something?
Original comment Steps Toward Super Intelligence Part I: How We Got Here
imgyuri
Neutral mention · rule
In terms of capturing the screen input for training, the fps is capped at 10 fps. This is following the number suggested in NVIDIA's paper[1], which cautions against overfitting. The program captures faster when the steering angle gets larger, because it suggests that curvy road is ahead, which are the more challenging scenarios for the program. In terms of testing, the max fps is determined by the performance of the GPU and the complexity of the model. I couldn't find any definitive studies done on the matter, but anywhere from 10-30 fps seems to work OK. [1] https://arxiv.org/abs/1604.07316
Original comment Europilot: A toolkit for Euro Truck Simulator to develop self-driving algorithms
tansey
Neutral mention · rule
> The data is useless unless it is annotated (e.g. a human labels where the lanes are, where the obstacles are, what the bicylist is doing, etc.) - that's the bottleneck, not collecting large amounts of raw sensor data + driver actions. Except that is exactly what Nvidia did, and it worked out fine for them: https://arxiv.org/abs/1604.07316
Original comment Master Plan, Part Deux
edshiro
Neutral mention · rule
The steering angles are probably automatically obtained from the script you run when manually training the toy vehicle. I presume at every frame an image is captured along with the steering angle for this image. You then create a neural net architecture that is being fed images and steering angle positions and outputs steering angle predictions. This is a regression problem that can be expressed in plain English as: "First train the neural network with a collection of images and associated steering angles. After training, if I were to give the neural network a new image it has never seen before, what steering angle would the network predict?" NVIDIA has a paper on it[1], and I blog about a similar coursework I complete as part of Udacity's self-driving car nanodegree[2] [1] https://arxiv.org/abs/1604.07316 [2] https://towardsdatascience.com/teaching-cars-to-drive-using-...
Original comment A self-driving toy car using end-to-end learning
cr0sh
Neutral mention · rule
I think LIDAR will have a place, if they can get the per-sensor cost down to something reasonable (under $500.00 per sensor, for 3D 180 degree with at least 32 vertical res beams - or the equivalent). But I think first we'll see cars utilizing tech as described in this paper: https://arxiv.org/abs/1604.07316 ...and variations of it to handle other modeling and vision tasks. Self-driving vehicle systems are amazing complex; it won't ultimately be any single system or sensor, or piece of software or algorithm that solves the problem - it's going to be a complex mesh of all of them working in concert. And even then, there will be mistakes, injuries, and deaths unfortunately.
Original comment Tesla hires Andrej Karpathy
cr0sh
Neutral mention · rule
I doubt anything like that would happen. While Tesla Motors was founded prior to the creation of the Tesla GPU architecture, there's not really any overlap - in fact, I wouldn't be surprised if Tesla Motors wasn't using something like this from NVidia: http://www.nvidia.com/object/drive-px.html As far as any overlap software-wise is concerned, while it isn't super clear what Tesla Motors is doing for their self-driving systems, based on what I've seen it seems like they are using only "basic" lane-detection and identification along with some other algorithmic vision-based systems. I'm not saying that's everything they are doing, just what I have seen released publicly on…
Read the full comment
their vehicle platform. NVidia, on the other hand, has been experimenting with using neural networks (deep learning CNNs specifically) to drive vehicles using only camera information: https://arxiv.org/abs/1604.07316 This is actually a fun CNN to implement - I (and many others) implemented variations of it in the first term on Udacity's Self-Driving Car Engineer Nanodegree. We weren't told to do it this way, but I chose to do so after reviewing the various literature, plus it seemed like a challenge (and it was for me). Udacity supplied a simulator: https://github.com/udacity/self-driving-car-sim ...and we wrote code in Python (Tensorflow and Keras) to train and drive the virtual car. For my part, I had set up my home workstation with CUDA so that Tensorflow would utilize my GPU (a lowly GTX 750 TI SC - though it seems like it might have a similar GPU capability as NVidia's Drive-PX system, based on what I've researched - a Mini-ITX mobo, a PCI-E slot riser, and a GTX 750 would make a decent low-end deep-learning platform for self-driving vehicle experiments, and cost a fraction of what the Drive-PX sells for).
Original comment Volta: Advanced Data Center GPU

Keep following the thread.

More in Machine learning

More from the same topics.