Variational Continual Learning

Original paper by Cuong V. Nguyen, Yingzhen Li, Thang D. Bui and Richard E. Turner

Part 1. Paper Summary

1. Introduction

• Continual Learning

• Data continuously arrive in a possibly non i.i.d. way
• Tasks may change over time (e.g. new classes may be discovered)
• Entirely new tasks can emerge (Schlimmer & Fisher 1986; Sutton & Whitehead, 1993; Ring, 1997)

• Challenge for Continual Learning

• Balance between adapting to new data vs. retaining existing knowledge
• Too much plasticity → catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990; Goodfellow et al., 2014a)
• Too much stability → inability to adapt
• Approach 1: train individual models on each task → train to combine them (Lee et al., 2017)
• Approach 2: maintain a single model and use a single type of regularized training that prevents drastic changes in the influential parameters, but allow other parameters to change more freely (Li & Hoiem, 2016; Kirkpatrick et al., 2017; Zenke et al., 2017)

• Variational Continual Learning

• Merge online VI (Ghahramani & Attias, 2000; Sato, 2001; Broderick et al., 2013)
• with Monte Carlo VI for NN (Blundell et al., 2015)
• and include a small episodic memory (Bachem et al., 2015; Huggins et al., 2016)

2. Continual Learning by Approximate Bayesian Inference

• Online updating, derived from Bayes' rule

• Posterior after Tth dataset is proportional to the Posterior after the (T-1)th dataset multiplied by the likelihood of the Tth dataset

• Projection Operation: approximation for intractable posterior (recursive)

• This paper will use Online VI as it outperforms other methods for complex models in the static setting (Bui et al., 2016)

2.1. VCL and Episodic Memory Enhancement

• Projection Operation: KL Divergence Minimization

• 
• : normalizing constant (not required when computing the optimum)
• VCL becomes Bayesian inference if

• Potential Problems

• Errors from repeated approximation → forget old tasks
• Minimization at each step is also approximate → information loss

• Solution: Coreset

• Coreset: small representative set of data from previously observed tasks
• Analogous to episodic memory (Lopez-Paz & Ranzato, 2017)
• Coreset VCL: equivalent to a message-passing implementation of VI in which the coreset data point updates are scheduled after updating the other data
• : updated using and selected data points from (e.g. random selection, K-center algorithm, ...)
• K-center algorithm: return K data points that are spread throughout the input space (Gonzalez, 1985)

• Variational Recursion

• Algorithm

• Step 1: Observe
• Step 2: Update using and
• Step 3: Update (used for propagation)

• Step 4: Update (used for prediction)

• Step 5: Perform prediction

3. VCL in Deep Discriminative Models

• Multi-head Networks

• Standard architecture used for multi-task learning (Bakker & Heskes, 2003)
• Share parameters close to the inputs / Separate heads for each output
• More advanced model structures:
• for continual learning (Rusu et al., 2016)
• for multi-task learning in general (Swietojanski & Renals, 2014; Rebuffi et al., 2017)
• automatic continual model building: adding new structure as new tasks are encountered
• This paper assumes that the model structure is known a priori

• Formulation

• Model parameters
• Shared parameters: updated constantly
• Head parameter: at the beginning, updated incrementally as each task emerges

• For simplicity, use Gaussian mean-field approximate posterior:

• Network Training

• Maximize the negative online variational free energy or the variational lower bound to the online marginal likelihood with respect to the variational parameters

• : tractable / set as multivariate Gaussian (Graves, 2011; Blundell et al., 2015)
• : intractable → approximate by employing simple Monte Carlo and using the local reparameterization trick to compute the gradients (Salimans & Knowles, 2013; Kingma & Welling, 2014; Kingma et al., 2015)

4. VCL in Deep Generative Models

• Deep Generative Models

• Can generate realistic images, sounds, and video sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016)
• Standard batch learning assumes observations to be i.i.d. and are all available at the same time
• This paper applies VCL framework to variational auto encoders (Kingma & Welling, 2014; Rezende et al., 2014)

• Formulation - VAE approach (batch learning)

• : prior over latent variables / typically Gaussian
• : defined by DNN, , where collects weight matrices and bias vectors
• Learning : approximate MLE (maximize variational lower bound w.r.t. and )

• No parameter uncertainty estimates (used to weight the information learned from old data)

• Formulation - VCL approach (continual learning)

• Approximate full posterior over parameters:
• Maximize full variational lower bound w.r.t. and

• : task-specific → likely to be beneficial to share (parts of) these encoder networks

• Model Architecture

• Latent variables → Intermediate-level representations

• Architecture 1: shared bottom network - suitable when data are composed of a common set of structural primitives (e.g. strokes)
• Architecture 2: shared head network - information tend to be entirely encoded in bottom network

5. Related Work

• Continual Learning for Deep Discriminative Models (regularized MLE)

• ML Estimation - set

• MAP Estimation - assume Gaussian prior and use CV to find → catastrophic forgetting

• Laplace Propagation (LP) (Smola et al., 2004) - recursion for using Laplace's approximation
• Diagonal LP: retain only the diagonal terms of to avoid computing full Hessian

• Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) - modified diagonal LP
• Approximate the average Hessian of the likelihoods using Fisher information
• Regularization term: introduce hyperparameter, remove prior, regularize intermediate estimates

• Synaptic Intelligence (SI) (Zenke et al., 2017) - compute using importance of each parameter to each task

• Approximate Bayesian Training of NN (focused on )

|Approach|References| |-|-| |extended Kalman filtering|Singhal & Wu, 1989| |Laplace's approximation|MacKay, 1992| |variational inference|Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011; Blundell et al., 2015; Gal & Ghahramani, 2016| |sequential Monte Carlo|de Freitas et al., 2000| |expectation propagation|Hernández-Lobato & Adams, 2015| |approximate power EP|Hernández-Lobato et al., 2016|

• Continual Learning for Deep Generative Models

• Naïve approach: apply VAE to with parameters initialized at → catastrophic forgetting
• Alternative: add EWC regularization term to VAE objective & approximate marginal likelihood by variational lower bound
• Similar approximation can be used for Hessian matrices for LP and → for SI (Importance sampling: Burda et al., 2016)