$$ \newcommand{\pmi}{\operatorname{pmi}} \newcommand{\inner}[2]{\langle{#1}, {#2}\rangle} \newcommand{\Pb}{\operatorname{Pr}} \newcommand{\E}{\mathbb{E}} \newcommand{\RR}{\mathbf{R}} \newcommand{\script}[1]{\mathcal{#1}} \newcommand{\Set}[2]{\{{#1} : {#2}\}} \newcommand{\argmin}[2]{\underset{#1}{\operatorname{argmin}} {#2}} \newcommand{\optmin}[3]{ \begin{align*} & \underset{#1}{\text{minimize}} & & #2 \\ & \text{subject to} & & #3 \end{align*} } \newcommand{\optmax}[3]{ \begin{align*} & \underset{#1}{\text{maximize}} & & #2 \\ & \text{subject to} & & #3 \end{align*} } \newcommand{\optfind}[2]{ \begin{align*} & {\text{find}} & & #1 \\ & \text{subject to} & & #2 \end{align*} } $$

Gatys et al. present a way to repurpose a deep network trained for image
classification to redraw images in the style of some reference image.
For example, this method can be used to render arbitrary photographs in the
style of Van Gogh’s *The Starry Night*.

Here’s a popular open-source implementation of the paper that includes some visual examples: https://github.com/jcjohnson/neural-style.

The problem is defined by two images: a *content* image and a *style* image.
The goal is to generate a *blended* image that resembles the *content* image
drawn in the fashion of the *style* image. Here’s an example from the paper:

In the above picture, the photograph of the waterfront is the content image,
*The Starry Night* is the style image, and the starry-night waterfront is the
blended image.

Let denote the content image, the style image, and the blended image, and let be the representation of in the -th layer of the network (similarly for and ). The blended image is found by approximately solving the below optimization problem.

where is fixed, is a weight that incorporates the size of layer (see the paper for details), and is the optimization variable. Gram matrices are used above to capture correlations between the features within each layer. The -weighted expression measures the similarity to the content image, while the -weighted term measures the similarity to the style image.

In the paper’s notation, corresponds to , corresponds to (the gram matrix) of , and corresponds to .