So Craig recently asked about artistic style transfer in another discussion on the main forum. I thought about it a little bit, and let's dive into it. What i want to start out with is a lecture from a Stanford neural net course from 2017. The lecture is on Feature Visualization in deep learning neural networks.
Now, a lot of what is discussed in this lecture is going to be way over the heads of most people who frequent our user forum, since they are artists, not deep learning practitioners or computer scientists. But that's ok, because this particular lecture is very visual. So it's ok if the specifics of what is discussed is over your head, just concentrate on the pictures being shown. Because all of you are very visually oriented, in an artistic way.
The lecture starts off running through the history of feature visualization. Feature visualization is an approach to try and understand what is going on inside of a deep learning neural network. For now, just accept that a deep learning neural network is composed of a bunch of different layers. Each layer has associated features. This research is trying to understand how these layer features relate visually to the task the neural network has been trained on.
Typically the neural networks used were trained for image classification. What this means is that the neural network looks at an image, and then tries to tell you what is in that image (classification). So if you had an image of a cat, then the neural network would ultimately return something that corresponds to a label for 'cat'.
The learning process starts off with a big database of different images with different classes of objects in it. So, imagine a database with 1000 different classes of objects (cat, dog, chair, table, etc). Each class might have 100 different images in it. So there would be a folder marked cat, and then in that folder would be 100 different images of different cats. Continue on like that for all 1000 different classes.
This image database is then used to train the neural network. Don't worry about how that works, other than being aware that the features associated with each layer are being adjusted during the training process. At the end, magically the neural network can distinguish between all of the different categories when feed a particular image as it's input.
At that point the neural network is trained. So you can then use it to classify any image. New images that are not in the original database.
Hope that made sense. Again, don't worry about the details on how it works, just be aware that there are different layers, with associated features, and these features are going to correspond visually to different properties of the images.
Now after running through the history of feature visualization, with associated output imagery that get's better and better over time, Justin then pivots to discussing the history of texture synthesis, neural network texture synthesis, and then artistic style transfer.
All of this stuff is related. And again, the technical details are going to be over your head. Just focus on the images being generated.
Again, the artistic style transfer stuff is at the end of the lecture, but the stuff leading up to that is the key to understanding what is going on in neural network style transfer. And why i mentioned in the previous discussion that you can think about it as a more elaborate kind of texture synthesis. Focus on how the imagery keeps improving over time as we move through the lecture.
With that said, on to the lecture.
You can watch it here.
Replies