Title: Mathematics of deep neural networks
Instructor:Mohammad Motamed
General description
We will study several concepts related to deep learning from an applied mathematics perspective. We will review recent articles on the subject. If needed, we will also be working with Python-Keras for implementation. We will be covering all or a portion of the following topics.
- Deep Neural Networks: formalization and key concepts
- What is a neural network? A parametric map with a compositional structure.
- What is the use of a neural network? It solves regression and classification problems.
- What is network training? It is an optimization problem.
- How to solve the optimization problem? By (stochastic) gradient descent & back propagation.
- Convergence of (stochastic) gradient descent.
- Choices of loss functions and activation functions
- Regularization
- Good practices for training neural networks.
- Approximation theory for neural networks
- Density: the theoretical ability to approximate well (with very large number of parameters)
- Convergence rates: how well is the approximation for a fixed number of parameters?
- Complexity: how many parameters needed to achieve a desired accuracy?
- Linear vs nonlinear methods of approximation
- Residual networks (ResNets) and residual Fourier feature networks (ResFFNets)
- Convolutional networks
- Deep networks for solving high-dimensional PDEs
Reading list
1) HH: C F Higham and D J Higham. Deep learning: An introduction for applied mathematicians. SIAM Review 61 (2019), pp. 860-891. 2) P: A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica 8 (1999), pp. 143-195. 3) Y: D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks 94 (2017), pp. 103-114. 4) BCN: L Bottou, F E Curtis, and J Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review 60.2 (2018), pp. 223-311. To be completed ... Grading: In-class participation and a final project motamed@unm.edu
Last updated: Spring 2023