Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik
Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
2020 · arXiv
We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(θ, φ)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
8 selected excerpts · 8 total mentions
Curious about the actual architecture. From the outside it looks like Gaussian splatting anchored to roughly one viewpoint, since the moment you wander outside the original frame or behind an object, it becomes messy. But Ben Mildenhall is one of the co-founders and a NeRF co-author (https://arxiv.org/abs/2003.08934), so I'm betting that whatever they're doing is more interesting than naive splatting. Curious if OP can share anything about the pipeline.
This apparently doesn't apply here, but in fact, pixels can be generated independently of each other. There are architectures where you can generate an arbitrary pixel or element of the image without generating the others; they are just implicit. See NeRFs or 'single-pixel GANs' or MAEs: eg https://arxiv.org/abs/2003.08934 https://arxiv.org/abs/2011.13775 https://arxiv.org/abs/2401.14391 Why is this possible? I tend to think of it as reflecting the ability to 'memorize' all possible data, and the independent generation is just when you 'remember' a specific part of a memory. The latent space is a Platonic object which doesn't change, so why should your generative process…
for materializing any specific point in the latent space have to? It's not surprising if you could generate arbitrary points from a function like 'y = mx + b' without generating every other point, right? It's just an atemporal mathematical object. Similarly with 'generating images from a random seed'. They too are just (complicated) functions mapping one number to another number. (You might wonder if this is limited to images? It is not. In fact, you can generate even natural language like this to some degree: https://github.com/ethan-w-roland/AUNN based on my proposal for taking the 'independent generation' idea to a pathological extreme: https://gwern.net/aunn )
NeRF was already standardized in the literature by people who are not these authors [0] and HDR is also already standardized by people who are not these authors [1]. Go with the flow unless you have a very good reason not to. HDR is probably HDR and not some other clever name because there was no obvious clever name with an easy pronunciation. Whereas NRF is just so close to NeRF -- an actual word people are familiar with -- it's begging to be pronounced that way. [0] https://arxiv.org/abs/2003.08934 [1] https://en.wikipedia.org/wiki/High_dynamic_range
At least in the original paper [1], the key idea is that instead of training a neural network to treat "images" as an array of pixels, you train a network to map a camera location and direction (in 3D space) to a distance and a color. For example, if you tell a NeRF that you have a camera at location (x, y, z) and pointing in direction (g, h, j), then the network will output the distance at which the ray emitted from the camera is expected to "hit" something and the RGB color of whatever is hit. Doing things in this way enables rendering images at arbitrary resolutions (though rendering can be slow), and is naturally conducive to producing rotated views of objects or exploring 3D space. Also, at least theoretically, it should allow for more "compact" network architectures, as it does not need to output, say, a 512x512x3 image. [1] https://arxiv.org/pdf/2003.08934.pdf
... that one has 52K citations, and the 2D to 3D paper "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" with 1488 citations. https://arxiv.org/abs/2003.08934
For some additional context, when the original NeRF paper (https://arxiv.org/pdf/2003.08934.pdf) was published 2 years ago, it reportedly took at least 12 hours (depending on hardware used of course) to train on the scene with the bulldozer. This has now been reduced to about 5 seconds (!), with realtime rendering of the result.
> If you're asking me for an example of it being publicly used in production, well I think you're asking a lot considering the technique is only a few months old That is what I explicitly asked for. You’re failure to provide an example is not because it’s new it’s because it’s actually not useful practically at the moment. NeRF has been around since March 2020 (https://arxiv.org/abs/2003.08934); you are simply wrong; traditional techniques are better right now, have better implementations and are widely used. NeRF is a promising technology that is categorically worse in its current implementation and maturity. I don’t know what else to say.
I'm no data scientist but many statistical methods rely on prior knowledge and even computed inputs. Two examples I can think of are doing linear regression on the square of your input. For deep learning, people have improved visual representation by taking samples of the colors at various frequencies. [1] [1]: https://arxiv.org/pdf/2003.08934.pdf
More from the same topics.
William P. Thurston
In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.
Tim Kraska, Alex Beutel, Ed H. Chi +2
Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.