ANC Workshop: Lukasz Romaszko and Nathalie Dupuy, Chair: Yuanhua Huang

Lukasz Romaszko

"Prediction and Explanation of Multiple Objects via Vision as Inverse Graphics"

The goal of computer vision is to understand images, so to infer the objects present in a scene, their poses, fine-grained details, as well as to understand the relations between objects. To this end, numerous approaches have been proposed, and a wide range of image interpretation tasks have been designed to evaluate their quality.
Although several methods for object detection have been developed, they are usually limited to a prediction of a 2D bounding-box in the image frame, without any richer description of the object shape or their relation in space to other objects.

Another way to address the recognition problem is to consider a 3D scene representation and its projection into the image. Scenes can be generated using a graphics renderer, the vision task then becomes the inversion of the rendering process.  We are interested in explanation of the whole scene configuration, so we aim to predict the camera pose, the illumination and the description of the objects present in the scene.

To start with, we consider simple scenes of various items located on a tabletop. We make use of a stochastic scene generator, which renders scenes to produce images. Images with labels serve as an input for training of the recognition model, so all the latent variables are directly available. We plan to evaluate our method on realistic and real images.

Nathalie Dupuy

"Interplay between off-line replay and plasticity in memory consolidation"

New declarative memories are initially stored in the hippocampus, and are subsequently slowly reorganized into semantic networks in the neocortex. It is held that this transition, called systems consolidation, is mediated by internally driven (off-line) replay of the memory traces. An emerging hypothesis is that the salience of an experience can modulate memory reactivation during off-line replay, and hence can impact systems consolidation.

Such ideas have been successfully applied to Reinforcement Learning algorithms. Here we hope to gain insights into the biological implementation of such algorithms by investigating whether selective off-line replay can account for rodent experiments. Particularly, we study the extraction of regularities during training in a water-maze memory task.

We implement a hierarchical computational model of hippocampal-neocortical interaction for learning and memory consolidation. The model is composed of (1) a Restricted Boltzmann Machine (neocortex) to extract semantic information and (2) a hippocampal module to support rapid memory encoding and recall. The hippocampus drives the off-line replay during sleep. Within this framework, we study how selective off-line replay alters the creation of semantic networks in the neocortex. We then investigate the adaptation of semantic information to novel experiences.