This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthezing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Great work! The fact that it captures "long-range order" seemingly perfectly is something not many have been able to do before! And the "collapse" visualization is great fun to watch.
But is your algorithm really qualitatively all that different from previous search methods (e.g. Efros and Leung), if you are still (uniform random?) sampling over the input distribution of patches?
I notice also your input textures tend to have sharp boundaries (as is common in pixel art). It would be interesting to see the results when the input has a lot of "noise", such as high def images of rocks or clouds ;)
While I still prefer search methods because they are easy to implement (and give better results), "deep" methods are definitely gaining some ground:
Deep Textures
http://bethgelab.org/deeptextures/
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
https://arxiv.org/abs/1601.04589
Take a look at these:
* Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis (http://arxiv.org/abs/1601.04589)
* A Neural Algorithm of Artistic Style (http://arxiv.org/abs/1508.06576)
They do much more than photoshopping. It's actual synthesis of new images in the style of a desired painter.
According to their about page, that site is using this algorithm http://arxiv.org/pdf/1508.06576.pdf whereas neural-image-analogies is a combination of http://www.mrl.nyu.edu/projects/image-analogies/index.html and http://arxiv.org/pdf/1601.04589v1.pdf
Basically you can do a few other things with image analogies, such as texture-by-numbers, that the neural style algo doesn't cover. In addition, the matching and blending covered in the Li and Wand paper provides more accurate style details.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.