Skip to content
Back to the collection

A Neural Algorithm of Artistic Style

Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

2015 · arXiv

Read the paper
HN score
0
Mentions
21
Readers
19
Months
18
What’s in a score?

From the abstract

In fine art, especially painting, humans have mastered the skill to create unique visual experiences through composing a complex interplay between the content and style of an image. Thus far the algorithmic basis of this process is unknown and there exists no artificial system with similar capabilities. However, in other key areas of visual perception such as object and face recognition near-human performance was recently demonstrated by a class of biologically inspired vision models called Deep Neural Networks. Here we introduce an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality. The system uses neural representations to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images. Moreover, in light of the striking similarities between performance-optimised artificial neural networks and biological vision, our work offers a path forward to an algorithmic understanding of how humans create and perceive artistic imagery.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 21 total mentions

wizzwizz4
Neutral mention · rule
> In September 2019 this must have been a GAN face. Well, yes (that file was), but actually no. StyleGAN1's public release was February 2019, and it's capable of far more than just faces. > Three years ago there was nothing remotely as impressive as modern GPT style I predicted that! Albeit, not publicly. (My predictions claimed it would have certain limitations; I can show that those still exist in GPT-4, but nobody on Hacker News seems to understand when I try to communicate it.¹) > or text to image models. Artbreeder (then called Ganbreeder) existed in early 2020, and it didn't take me by surprise when it came out. It parameterises the output of the model by mapping…
Read the full comment
sliders to regions of the latent space; quite an obvious thing to do, if you want to try getting fine-grained control over the output. (A 2015 paper built on this technique: https://arxiv.org/abs/1508.06576) I was using spaCy back in 2017–2018-time. It represents sentences as vectors, that you can do stuff like cosine similarity on. If I'd been more interested in the field back then, I could have put two and two together, and realised you could train a net on labelled images (with supervised learning) to map a spaCy model's space to StyleGAN's, which would be a text-to-image model. It was very much imaginable back before April of 2020; a wealthy non-researcher hobbyist could've made one, using off-the-shelf tools. If I were better at literature searches, I could probably find you an example of someone who'd done that, or something like it! --- ¹: See e.g. here: they tell me GPT-4 can translate more than just explicitly-specified meaning, and the "evidence" doesn't even manage that. https://news.ycombinator.com/item?id=35530316 (They also think translating the title of a game is the same as translating the game, for some reason; that confusion was probably my fault.)
belter
Neutral mention · rule
That is one of the two concerns. The other concern, is that will be a paid product. It's users decide to pay or not, based on the quality of the code suggestions. The quality of the code suggestions are tied to the quality of the training corpus. The manufacturer of the product needs to get permission, explicitly, or implicitly via a license associated with the code corpus, to use the training set on the building of the commercial product. Speaking exclusively for US jurisdictions, so far courts seem to have been accommodating to the so called discriminative algorithms, but the jury is still out, for the so called generative algorithms. [1] Instead of examples like…
Read the full comment
Copilot or Codewhisper producing code, let's think of examples like some of recent advances using Convolutional Neural Networks [2] What will the Music Industry or Cinema Industry do, if a company uses the Corpus of Music or Movies without asking for further permission? For example, to produce a commercial product, that would allow a user, to create music on the style of Pat Metheny, AC/DC, Abba, or a Cinematic experience on the style of Lord of the Rings? Surely both the user and the product vendor would be called upon it? [1] "The Most Important Court Decision For Data Science and Machine Learning": https://towardsdatascience.com/the-most-important-supreme-co... [2] https://arxiv.org/pdf/1508.06576v1.pdf
bpizzi
Neutral mention · rule
> I really don't see how machines could distill the ideas out of a text and reinterpret them in a nice way. Well, machines are already doing it with paintings: https://arxiv.org/pdf/1508.06576v2.pdf. There's no technical reason for not being able to achieve this sort of 'machine creativity' with texts. It's just a harder version of the same problem. > I can see AI work for short, simple, well-written texts. Heck, even long manuals with repetitive blocks of text from manual to manual. But with creative, complex text: still a looooong way baby! I (carefully, waiting for experts' feedback) disagree. ML works on data and algorithms: give a deep RNN a great lot of human made…
Read the full comment
translations, time/cpu to produce a good model, and I think you would be astonished by the results. Data's complexity is of little hindrance here, because the algorithm does not try to understands the data (as we humans like to think we're doing when reading), it's just trying to infer a solution based on learned behaviors. And it's 'just' ML, AI will come when your computer tell you that this specific text makes it sad, for example.
dbranes
Neutral mention · rule
Ah okay right this is the answer. Previous approaches [1] are deep generative models that you have to optimize for each input, whereas here you run just a forward evaluation on a model that you've trained beforehand. I would still argue the term closed-form is misleading here, because: - Even during training at any given time you can read off a "closed-form expression" of the neural network of this type, so closed-form in this broad sense really doesn't mean much. Furthermore any result of any numerical computation ever are also closed-form solutions according to this, on the grounds that they result from a computation that completed in finite number of steps. So really…
Read the full comment
whenever you ask a grad student to run some numerical simulation expect them to come back saying "Hey I found a closed-form expression!" - The reason the above is absurd is that these trained NN's aren't really solutions to the optimization problem, but approximations. So this is really saying I have a problem, I don't know how to solve it but I can produce a infinite sequence of approximations. Now I'm gonna truncate this sequence of approximations, and call this a closed form solution. The analogy in highschool math would be computing an infinite sum that doesn't converge, but now let's instead just add to some large N, and call this a closed-form solution. [1] e.g. https://arxiv.org/pdf/1508.06576.pdf
joshvm
Neutral mention · rule
Absolutely you can do 'AI Research' without a degree in maths. Sure you need a grounding in linear algebra, stats, probability and calculus, but not much more than a CS or physics degree will teach you. That stuff is indeed learned easily, and it's not what the Reddit user is worried about. That also ignores applications of machine learning, which is also a massive (and lucrative) field. But because it's a trendy field, I think there is an obsession with people needing to understand everything theoretical that comes out for fear of missing the boat. Some of the really interesting papers that have come out over the last few years - for example artistic style transfer and…
Read the full comment
Faster R-CNN - have hardly any maths in. You can count the equations on one hand in both those papers. No doubt the authors know their stuff, but how readable are those papers compared to e.g. a 100-page proof? Which did I learn more from? They're a combination of two things: intuitive network architecture and a clever loss function. The first thing is a combination of intuition and programming, the second involves a little maths, but nothing outrageous. [1] https://arxiv.org/abs/1508.06576 [2] https://arxiv.org/abs/1506.01497
radarsat1
Neutral mention · rule
> If you succed in manufacturing a stigma against arXiv papers That's not exactly my intention, and your statement goes in line with what I said.. > sure I've cited software in the past, and having an article to go with it is even better where "software" could also include blog posts. It's just that, yeah, when it comes to it if I have to choose between citing something some guy wrote on a blog or citing Arxiv, I'll choose the latter every time. But my question was not about whether arxiv is an appropriate medium for publishing ideas, I'm certainly not arguing that it should go away, but rather whether it's an appropriate thing to be citing in a scientific context. i.e.…
Read the full comment
in derived work.. should it be "okay" for people to publish to arxiv and just.. leave it there and not put it in a conference or journal? Should such work be validated by citation? It's a legitimate question that I don't know the answer to. If you get an idea from there, you can't just.. not cite it.. and yet, I feel like arxiv should be used only as long as the work will eventually get a proper publication. Which more and more it seems is not a given. I'm really just responding to the statement on the posted link, which is just a header, "Citation", and an Arxiv-bibtex entry. No "submitted to X..", "in press", etc., or anything. Again, it's not that I'm totally against the idea, but I think there must be some happy medium between "peer reviewed work" and "well-written but unreviewed article". Typically this used to be conferences, but conferences are expensive, and with people uploading their pre-conference unaccepted publications and those getting citations, I mean.. where will this end? I was a bit shocked recently while writing an ML article to find that the "official" original reference, that you see cited everywhere, for the whole concept of "style transfer" appears to be an arxiv paper [1]. My reservation doesn't so much come from the fact that the authors put their work there, but more that this is what people are actively choosing to cite over their peer-reviewed conference publication [2]. When does "pre-print" become just "print"? (Currently Google Scholar reports 216 citations for the former and 74 citations for the latter. Of course they were published a few months apart so it's not a great comparison but still, just an example..) [1]: https://arxiv.org/abs/1508.06576 [2]: http://www.cv-foundation.org/openaccess/content_cvpr_2016/ht...
jthickstun
Neutral mention · rule
Hi savanaly, When we talk about "what makes Bach sound like Bach," the technical concept we have in mind is the recent work in computer vision on style transfer. For example, https://arxiv.org/abs/1508.06576 We are excited to work on adapting these models to the musical domain! As for note prediction, you can see our results in our paper: https://arxiv.org/abs/1611.09827 Our are results for simple (2-layer, not very "deep" models); we were interested in understanding the low-level "features" of music rather than building a model that maximizes performance. Nevertheless, the results are quite promising; I'm confident that someone using our dataset with a deep network and a lot of gpus could blow our numbers out of the water! :) Tutorials on how to set up and evaluate this task are available on our website: http://homes.cs.washington.edu/~thickstn/start.html
n17r4m
Neutral mention · rule
Maybe this?: https://www.youtube.com/watch?v=Rfhxy2w3Qi4 Using a gram matrix across several layers to hold the style, and more conventional activation optimization for the content. Trained on VGG. It combines the two loss functions of style and content to optimise noise towards a output image. The amazing thing is being able to mathematically separate content from style; in a similar fashion to how the brain can perform this task. See: https://arxiv.org/abs/1508.06576 Fwiw, I'm part of a group trying a similar thing on very deep residual net's with recurrent net's to improve upon the temporal sweeps.
andreyk
Neutral mention · rule
They say they use a variation on the neural art transfer algorithm: "We apply an algorithm developed by Leon Gatys, Alexander Ecker and Matthias Bethge. " https://deepart.io/page/about/ https://arxiv.org/abs/1508.06576 There are various tweaks for how to do it and some hacks to make it faster, so they probably iterated to get better results.
erkaman
Neutral mention · rule
The demo uses WebGL, and if you can't get the demo to work, you can here(https://github.com/Erkaman/regl-cnn) find a recorded gif that shows what it is supposed to look like. This demo does handwritten digit recognition by evaluating a Convolutional Neural Network on the GPU with WebGL. The network was trained in TensorFlow by this script here(https://github.com/Erkaman/regl-cnn/blob/gh-pages/scripts/cr...), and the network was then reimplemented on the GPU by hand with WebGL. The main purpose of the demo was to demonstate how our WebGL framework regl(https://github.com/mikolalysenko/regl) can be used to greatly simplify GPGPU programming in WebGL. The secondary purpose…
Read the full comment
was to test whether evaluating Deep Learning networks in WebGL is doable. To our knowledge(but we may be wrong!), our implementation is the first implementation ever to attempt GPU accelerating neural networks with WebGL And we hope that this implementation will provide a foundation for people who, like us, wish to experiment with Deep Learning and WebGL The GPU implementation can be found here(https://github.com/Erkaman/regl-cnn/blob/gh-pages/src/gpu.js) Note that this network will probably be slower than the corresponding network implemented on the CPU. This is because of the overhead associated with transferring data to and from the GPU. But in the future we will attempt implementing more complex networks in the browser, such as Neural Style(https://arxiv.org/pdf/1508.06576v2.pdf), and then we think that we will see a significant speedup compared to the CPU. Lastly, if anyone has any questions, I will be glad to answer them here.
awentzonline
Neutral mention · rule
According to their about page, that site is using this algorithm http://arxiv.org/pdf/1508.06576.pdf whereas neural-image-analogies is a combination of http://www.mrl.nyu.edu/projects/image-analogies/index.html and http://arxiv.org/pdf/1601.04589v1.pdf Basically you can do a few other things with image analogies, such as texture-by-numbers, that the neural style algo doesn't cover. In addition, the matching and blending covered in the Li and Wand paper provides more accurate style details.
DanielleMolloy
Neutral mention · rule
Note that all of the computer generated pieces were created by this impressive deep net algorithm (paper prepublished in 8/2015): http://arxiv.org/abs/1508.06576 Somehow I cannot believe how fast they got the idea to create profit from it: http://deepart.io/
temuze
Neutral mention · rule
One weird trick for parallelizing ConvNets: http://arxiv.org/abs/1404.5997 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks: http://arxiv.org/abs/1511.06434 A Neural Network of Artistic Style: http://arxiv.org/abs/1508.06576
nl
Neutral mention · rule
Not really sure why this page made it to HN. This group's most interesting work was the paper that outlines "style transfer"[1], which is what all those photos-painted-in-the-style-of-van-gogh-etc pictures[2] that went around a few months ago were using. [1] http://arxiv.org/abs/1508.06576 [2] eg, my pretty average effort: https://twitter.com/nlothian/status/646280514484043776
deepnet
Neutral mention · rule
http://arxiv.org/abs/1508.06576 [non - pdf] Here we introduce an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality. The system uses neural representations to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images Gatys, Echer, & Bethge
erkaman
Neutral mention · rule
The demo uses WebGL, and if you can't get the demo to work, you can here(https://github.com/Erkaman/regl-cnn) find a recorded gif that shows what it is supposed to look like. This demo does handwritten digit recognition by evaluating a Convolutional Neural Network on the GPU with WebGL. The network was trained in TensorFlow by this script here(https://github.com/Erkaman/regl-cnn/blob/gh-pages/scripts/cr...), and the network was then reimplemented on the GPU by hand with WebGL. The main purpose of the demo was to demonstate how our WebGL framework regl(https://github.com/mikolalysenko/regl) can be used to greatly simplify GPGPU programming in WebGL. The secondary purpose…
Read the full comment
was to test whether evaluating Deep Learning networks in WebGL is doable. To our knowledge(but we may be wrong!), our implementation is the first implementation ever to attempt GPU accelerating neural networks with WebGL And we hope that this implementation will provide a foundation for people who, like us, wish to experiment with Deep Learning and WebGL The GPU implementation can be found here(https://github.com/Erkaman/regl-cnn/blob/gh-pages/src/gpu.js) Note that this network will probably be slower than the corresponding network implemented on the CPU. This is because of the overhead associated with transferring data to and from the GPU. But in the future we will attempt implementing more complex networks in the browser, such as Neural Style(https://arxiv.org/pdf/1508.06576v2.pdf), and then we think that we will see a significant speedup compared to the CPU. Lastly, if anyone has any questions, I will be glad to answer them here.

Keep following the thread.

More in Algorithms

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

16 mentions14 months
2.5HN score