Course Information
Description
This seminar is aimed at Master’s students of Scientific Computing and Mathematics. Participants require basic knowledge about neural network architectures (e.g., fully-connected feedforward and residual networks, nothing too advanced), neural network training and basic programming knowledge, e.g., in Python. We view Deep Learning from a mathematical point of view and discuss how we can relate it back to well studied mathematical concepts. This semester the focus topic is „Challenges in Deep Learning". Together we explore various aspects of the selected papers, e.g. content, authors, references, related research, etc.
The seminar will be held in a role-playing style, similar to what is described in https://colinraffel.com/blog/role-playing-seminar.html. We have selected five roles. More details on the roles will be announced during the preparatory meeting. Each participant will be assigned a number and will fill each role one time according to the schedule.
Topics
In every meeting we will study one paper addressing a challenge in deep learning.
- Glorot, Bengio: Understanding the difficulty of training deep feedforward neural networks
- Ba, Caruana: Do deep nets really need to be deep?
- Zhang, Bengio, Hardt, Recht, Vinyals: Understanding deep learning (still) requires rethinking generalization
- Li, Xu, Taylor, Studer, Goldstein: Visualizing the loss landscape of neural nets
- Grimmer: Provably faster gradient descent via long steps
Exam / Presentations
For a successful completion of the seminar it is mandatory to attend all sessions. The final grade will be a weighted sum of the grades of your presentations and of your participation in the subsequent discussions.
Dates and Timeline
- Organizational meeting: October 16th 2023 at 6pm in Mathematikon seminar room 11.
- Participation in this meeting is mandatory to attending the seminar.
- During this meeting, we will choose the regular seminar dates from the options below, and the roles will be assigned.
- Due to the interactive style of the seminar it is mandatory to attend every meeting in person. There are going to be five 90-minute-sessions distributed over the semester, starting in the beginning of November.
- If you have questions, please don’t hesitate to contact us.
Meeting dates are:
- Tuesday 18:00-19:30 (31.10., 21.11., 05.12., 19.12., 09.01.) in Mathematikon Seminar Room 2.
(Pre-)Registration
If you are interested in participating in this seminar, please pre-register via Müsli, so we know how many participants we can expect. You will finally register for the class when you have been assigned a role number. We reserve the right to select students according to degree program, previous knowledge, and other criteria.