Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras

solved because we can by now do unsupervised training and achieve usable results. While I had trouble reproducing this specific paper, the general idea is valid: https://github.com/google-research/google-research/tree/mast... https://arxiv.org/abs/1904.04998 We are also seeing good results from using random YouTube videos to train AI vision. But given that you are so sure that this is unsolved, I wonder if I should start a company to sell my depth estimate pipeline. Would you have any example image pairs that are causing problems, so that I can see visually what fails?

example self-supervised learning from video. A great demonstration of the usefulness of self-supervision is monocular depth estimation: taking consecutive frames (2D images) we can estimate per pixel depth and camera ego-motion by training to wrap previous frames into future ones. The result is a model capable of predicting depth on individual 2D frames. See this paper [1][2] for example. By using this kind of approach, we can lower the need for precisely annotated data. [1] https://arxiv.org/abs/1904.04998 [2] more readable on mobile: https://www.arxiv-vanity.com/papers/1904.04998/ Edit: typo

(rather than its quality) accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations --- essentially mimicking LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance --- raising the detection accuracy of objects within 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo image based approaches. Karpathy was also pointing to: Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras https://arxiv.org/abs/1904.04998 We present a novel method for simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos, using only consistency across neighboring video frames as supervision signal. Similarly to prior work, our method learns by applying differentiable warping to frames and comparing the result to adjacent ones, but it provides several improvements: We address occlusions geometrically and differentiably, directly using the depth maps as predicted during training. We introduce randomized layer normalization, a novel powerful regularizer, and we account for object motion relative to the scene. To the best of our knowledge, our work is the first to learn the camera intrinsic parameters, including lens distortion, from video in an unsupervised manner, thereby allowing us to extract accurate depth and motion from arbitrary videos of unknown origin at scale. We evaluate our results on the Cityscapes, KITTI and EuRoC datasets, establishing new state of the art on depth prediction and odometry, and demonstrate qualitatively that depth prediction can be learned from a collection of YouTube videos. Tesla has also advantage of having radars and is able to do supervised learning of depth estimates for moving objects from video. I was also skeptical of Tesla self-driving capability, because they had some stupid ideas, like ignoring radar data based on GPS tagging from fleet behavior. But lack of LIDAR will not be the issue. Not even close. It seems like they are focusing on building proper pipeline for training neural networks. The question is whether neural networks as a technology can handle self driving. Reasoning based on "human brain can do it, so artificial neural networks can do it" is wrong. Natural neural networks and artificial neural networks, besides name share only very, very rough low level conceptual ideas. Moreover, our ANN architectures are probably missing most of what brains do on high level. So, I think this is still an open question - can ANN do it? If not, then nobody will have full self driving capability widely deployed any time soon. Because, even tough neural networks are not perfect, everything else is super brittle in comparison. LIDAR and hires maps are technologies that give working short-term solution, are sort of local minimum. LIDAR is super expensive and already proven to be not necessary. While, hires maps are super brittle and too capital intensive to be widely deployed. But if neural networks are sufficient Tesla will leave everyone in dust. There will be literally no competition. They do not need to learn electric cars manufacturing in order to deploy their technology widely, they are doing that already. They will not need to backtrack on LIDAR and hires maps solutions. Also, their decision to deploy self driving hardware to every car means that they have access to stupidly big amount of real world data from all varieties of environments all around the world. BTW - Extrapolating based what Tesla was doing before Karpathy joined is probably misguided. I was afraid that he will get lost in a big corporation, but he seems to be doing great job there. In my opinion technology that build with his supervision will be significantly better than what Tesla was doing before. But Karpathy is not a magician. So, Tesla has the biggest potential and the big question is: Are neural networks sufficient for self driving?

loss function and issues with convergence, for example by fixing all random initialization to a seed and then parameter-scanning on that value (Very expensive). Notable examples of the difficulties faced would be https://arxiv.org/abs/1904.04998 which I have never seen converge from random initialization or https://arxiv.org/abs/1711.07837 which diverges without supervised pre-training. Also, while many people use the gaussian-distributed euclidean norm of the difference between prediction and groundtruth as their main loss term, there has been a lot of discussions recently if that is a good idea, because it forces the neural network to represent uncertainty with blur. But optical flow tends to have sharp edges where objects end. Combined, that means the problems that I work on usually do not have a gaussian-distributed loss and usually are irregular and never convex, so I'm not sure if I could use G-GLN. But the application to contextual bandits looks VERY interesting to me :) I see great potential in using conv layers as pre-compression and then applying decision trees on the resulting intermediate representation. What I previously did for object segmentation was to sample a random but fixed set of feature pair differences and use the signs as bit flags. I then trained the decision trees on those bits to predict the boundary shape around that pixel. I got the general idea from this paper: https://arxiv.org/abs/1406.5549 But it sounds like moving from difference bits to halfspaces and from linear regression to GLN, your paper "Online Learning in Contextual Bandits using Gated Linear Networks" could greatly improve on that. I'd be curious to see how those bandits do on BSDS500. BTW, are you aware of any discussion groups for AI image processing that are open to members of the public? https://www.reddit.com/r/deeplearning/ seems to be mostly people that just took a Udemy / Coursera course, so it's usually about re-using existing models and almost no talk about research.