Mohammad Motamed

MATH 557, Selected Topics in Numerical Analysis (graduate course)

Title: Mathematical foundations of deep learning

Instructor:Mohammad Motamed

General description

We will study several concepts related to deep learning from an applied mathematics perspective. We will review recent articles on the subject. We will also be working with Matlab and Python (Keras) for implementation. We will be covering the following topics.

  • Deep Neural Networks: formalization and key concepts
    • What is a neural network? A parametric map with a compositional structure.
    • What is the use of a neural network? It solves regression and classification problems.
    • What is network training? It is an optimization problem.
    • How to solve the optimization problem? By stochastic gradient descent & back propagation.
    • Choices of loss functions and activation functions
    • Regularization
    • Good practices for training neural networks.
  • Approximation theory for neural networks
    • Density: the theoretical ability to approximate well (when the number of parameters is very large)
    • Approximation rates: how well is the approximation for a fixed number of parameters?
    • Complexity: how many parameters needed to achieve a desired accuracy?
  • Optimization methods to train dee networks
    • The fundamentals of optimization for machine learning
    • When does gradient descent fail?
  • Deep networks for solving high-dimensional PDEs
    • How to solve a PDE by neurla networks?
    • Dimension independent approximability of neural networks

Reading list

1) HH:   C F Higham and D J Higham. Deep learning: An introduction for applied mathematicians. SIAM Review 61 (2019), pp. 860-891.

2) P:   A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica 8 (1999), pp. 143-195.

3) Y:   D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks 94 (2017), pp. 103-114.

4) BCN:   L Bottou, F E Curtis, and J Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review 60.2 (2018), pp. 223-311.

To be completed ...

Grading: In-class exercises and a final project
Last updated: Fall 2021