-
Notifications
You must be signed in to change notification settings - Fork 7
1708.05349
PixelNN: Example-based Image Synthesisn [PDF] [notes]
Aayush Bansal, Yaser Sheikh, Deva Ramanan
read 15/11/2017
Create plausible reconstructions from degraded representations of this image (for instance blurred, edges, surface normal map...)
Control image synthesis explicitly to produce, for instance by restricting the set of images it should resemble or by providing a guiding image.
A CNN is used to map from the degraded input to the reconstruction. This reconstruction is usually over-smoothed (reconstructs only low to middle frequencies)
The reconstruction is used to map each pixel to a pixel from a candidate image. This mapping takes into account different intermediate features extracted from a neural network. The cosine distances between features are used to evaluate the proximity between the context of the pixel of the intermediate reconstruction and the one of the training set. The mapping is then operated independently for each pixel. To speed-up the matching, K global nearest neighbours are first selected using high level features (from conv5 layer), and the pixel matches are looked for in a restricted window (not clear exactly how the center of the window is selected).
The output can be controlled by restricting the subset of the images in the training set which are used for the NN matching.
Looks similar in quality to Pix2Pix with different shortcomings (sometimes sharper results for PixNN)