580.691/491 Learning, Estimation, and Control
Textbook: Biological Learning and Control, Shadmehr and Mussa-Ivaldi, MIT Press.
The course introduces modern techniques in mathematical analysis of biomedical data, with a focus on learning and control. Techniques include maximum likelihood, estimation theory via Kalman equation, application of Kalman to Brain-Machine Interfaces, state-space models, Bayesian estimation, classification of labeled data, dimensionality reduction, clustering, expectation maximization, and dynamic programming and reinforcement learning via the Bellman equation. We will apply the techniques to better understand how the brain learns to control our behavior. This course is taught in the Winter/Spring semester.
Course topics and materials
-
Introduction
-
Regression and maximum likelihood
- Probability theory. Bayes rule, expected value and variance of random variables and sums of random variables, expected value of random variables raised to a power, Binomial distribution, Normal distribution.
- Least mean squared loss function, batch learning and the normal equation, cross validation.
- Newton-Raphson, LMS and steepest descent. Newton-Raphson, LMS and steepest descent with Newton-Raphson, weighted least-squares, regression with basis functions, estimating the loss function for learning in humans.
- Backpropagation.
- Examples of estimation and learning in biological systems (chapter 4.1–4.5). Maximum likelihood estimation; likelihood of data given a distribution; ML estimate of model weights and model noise, integration of multiple sensory data.
-
Estimation theory
- State estimation of dynamical systems (chapters 4.6 and 4.7). Optimal parameter estimation, parameter uncertainty, state noise and measurement noise, adjusting learning rates to minimize model uncertainty. Derivation of the Kalman filter algorithm.
- Kalman filter and signal dependent noise (chapters 4.9 and 4.10).
-
Bayesian estimation
- Applying Kalman to Brain Machine Interface. Bayes estimation and its relationship to Kalman filters (chapter 5.1). Factorization of joint distribution of Gaussian variables.
- Causal inference (chapters 5.2 and 5.3). The problem of deciding between two generative models.
- Examples from biological learning: Kamin blocking, backward blocking, and comparisons between Kalman learning and LMS.
- MAP estimation.
-
Classification with labeled data
- Fisher linear discriminant, posterior probabilities. Classification using posterior probabilities with explicit models of densities, kernel estimates of density.
- Logistic regression. Iterative re-weighted least squares estimation.
-
Dimensionality reduction and clustering
- Dimensionality reduction: Principal component analysis.
- Dimensionality reduction: t-SNE.
- Unsupervised classification: Mixture models, K-means algorithm, and Expectation-Maximization.
- EM and maximizing the complete log likelihood.
-
Bellman equation and optimal control
- Lagrange multipliers and constraint minimization (chapter 10).
- Open loop control with signal-dependent noise (chapter 11).
- Introduction to optimal feedback control (chapters 12.1–12.3). Bellman equation.
- Optimal feedback control of linear dynamical systems with non-signal dependent noise.
- Optimal feedback control with signal dependent noise.
- Reinforcement learning.