MATH 557, Selected Topics in Numerical Analysis (graduate course)
Title: Mathematical foundations of deep learning
We will study several concepts related to deep learning from an
applied mathematics perspective. We will review recent articles on the
subject. We will also be working with Matlab and Python (Keras) for
We will be covering the following topics.
- Deep Neural Networks: formalization and key concepts
- What is a neural network? A parametric map with a compositional structure.
- What is the use of a neural network? It solves
regression and classification problems.
- What is network training? It is an
- How to solve the optimization problem? By stochastic
gradient descent & back propagation.
- Choices of loss functions and activation functions
- Good practices for training neural networks.
- Approximation theory for neural networks
- Density: the theoretical ability to approximate well
(when the number of parameters is very large)
- Approximation rates: how well is the approximation for a
fixed number of parameters?
- Complexity: how many parameters needed to achieve a desired accuracy?
- Optimization methods to train dee networks
- The fundamentals of optimization for machine learning
- When does gradient descent fail?
- Deep networks for solving high-dimensional PDEs
- How to solve a PDE by neurla networks?
- Dimension independent approximability of neural
1) HH:   C F Higham and D J Higham. Deep learning: An introduction for applied mathematicians. SIAM Review 61 (2019), pp. 860-891.
2) P:   A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica 8 (1999), pp. 143-195.
3) Y:   D. Yarotsky. Error bounds for approximations with
deep ReLU networks. Neural Networks 94 (2017), pp. 103-114.
4) BCN:   L Bottou, F E Curtis, and J Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review 60.2 (2018), pp. 223-311.
To be completed ...
Grading: In-class exercises and a final project
Last updated: Fall 2021