Nonlinear Optimization

Course Information

Description #

This class provides a detailed overview of algorithms for unconstrained and constrained smooth, nonlinear optimization problems in finite dimensions. It will emphasize the interplay of optimization and numerical linear algebra. Optimality conditions and associated constraint qualifications for constrained problems will be presented and discussed. Moreover, the class will give an introduction to algorithmic differentiation (AD).

Prerequisites #

Skills from Analysis I and II (MA1/ MA2), Lineare Algebra I (MA4) as well as basic optimization skills are highly recommended.

Concept and Components #

This class is comprised of:

• Detailed lecture notes
• 2x 2h lectures per week
• Exercise sheets with corresponding answer sheets
• Labs for co-working on exercise sheets
• An exam following the end of the lecture period

The detailed concept and how to activly participate in the class will be discussed in the first lecture on Monday 17.04.2023 at 11:15h in the main lecture hall of the Mathematikon.

• Plenum 00 (Informations Concerning the Class )

Dates and Schedule #

You can find the information on lecture meetings in the LSF. Information on the exercises can be found in Müsli.

Please also refer to the class’ schedule.

Exercises #

Exercises will be centered around one exercise sheet per week containing both theoretical/analytical questions and numerical/implementation exercises. Default programming language – in the sense that it will be used in our answer sheets and we will be providing support for it – will be Python. Feel free to work with another programming laguage that is suitable for implementing the optimization algorithms if you are self sufficient working with it.

In the exercise lab groups (see Müsli), you will have the oportunity to work on the exercise sheets with your peers and with support from our tutors. You can get the most out of their time if you show up having already taken a first look at the exercises and bringing your own questions and discussion topics to the meetings. Feel free to ask questions on the lecture notes or the lectures as well.

For successful participation in this semester’s exercises, you will be asked to submit your exercises weekly, see exams. To evaluate your success on the exercise sheets, please compare your work with the answer sheets we will provide and, of course, feel free to talk to your tutor in the labs.

You may work on the exercises in whichever type of group you would like but we request that you submit one answer/program per person.

Please submit your work via Moodle.

Dates and Schedule #

You can find the information on lecture meetings in the LSF. Information on the exercises can be found in Müsli.

Please also referr to the class’ schedule.

Attention: The class on 18.04.2023 will not take place in the lecture hall of the INF 205 but in seminar room 25 of INF 328.

Class Registration #

To register for this class, please register for an exercise group at Müsli. Please also register in case you are interested in participating in this class but not in the exercises. This will allow us to get an idea of class demand.

Attention(!): Registration for the class does not automatically include registration for an examination. More information on the exams can be found in the section exams.

Exams #

We will be offering exams at the end of the semester and during the following summer break.

Exam Format #

Exams will be held orally and you should plan for around 30-35 minutes of exam time plus a short additional debriefing. You may choose between english and german as the exam language.

You have the option of (orally and using a whiteboard) presenting a topic from the lecture’s content in (roughly) the initial 5 minutes of the exam, after which we will start the exam’s question-and-answer style discussion. We are offering the following choice of topics for your presentation (including the relevant sections from the lecture notes):

1. minimization of quadratic functions (all of §4)
2. gradient descent method with line search (§5.1, §5.2, §5.3)
3. inexact Newton method with line search (§5.4, §5.5, §5.6)
4. quasi-Newton method with line search (§5.4, §5.5, §5.7)
5. trust-region methods (all of §6)

Should you choose to take this opportunity, make sure that your chosen topic and the level of detail you go into match the short timeframe. If you choose to pass on this option, we will start out with the question-and-answer style discussion right away.

In any case, the exam will begin with a topic from chapter 1 (unconstrained optimization) but topics from chapters 2 and 3 (constrained optimization) will definitely be part of the exam as well. The proofs of the following lemmas and theorems may be part of the exam:

1. Theorem 3.1 (first-order necessary optimality condition)
2. Theorem 3.2 (second-order necessary optimality condition)
3. Theorem 3.3 (second-order sufficient optimality condition)
4. Lemma 5.5 (bounded condition numbers imply the angle condition)
5. Lemma 5.8 (efficiency implies admissibility)
6. Theorem 5.9 (global convergence of model Algorithm 5.2)
7. Theorem 7.4 (first-order necessary optimality conditions)
8. Lemma 7.7 (relation between the cones)
9. Lemma 8.4 (significance of the KKT conditions)
10. Theorem 8.7 (KKT conditions are necessary optimality conditions under the Abadie CQ and the Guignard CQ)
11. Theorem 8.9 (KKT conditions are necessary optimality conditions under affine constraints only)
12. Theorem 8.18 (KKT conditions are necessary optimality conditions under the LICQ)
13. Theorem 8.19 (KKT conditions are sufficient optimality conditions for convex problems)

Admission Rules #

If you want to take the exam with us, we require that you are registered in Müsli and that you meet at least one of the following criteria:

1. You have been admitted to an exam for this class in a previous semester and let us know via E-Mail when and by whom you have been admitted by 2023-06-30.
2. You have successfully participated in this semester’s exercise program in the sense that:
   • On every exercise sheet but at most one sheet, you have worked every exercise to a reasonable degree. “To a reasonable degree” means that you have at least sketched an approach of how to solve the excercise and made a connection to the material of the lecture.

Exam Dates #

The exams will be held in blocks of 3 hours covering 4 exams of 45 minutes per exam. You may request your preferred block for the exam and we will assign slots inside the blocks ourselves and let you know your assigned slot in the confirmation email.

The exam blocks are:

We will try to keep the current booking status up to date, but expect delays.

Exam Registration #

Please contact our front office via email providing the following:

• Your intent to register
• Your preferred exam block and possibly an alternative block
• Last name
• First Name
• Date of birth
• Place of birth
• Student-ID-Nr. (Matrikel-Nr.)
• Course of study

We require this information to pre-fill the exam forms.

Your registration is complete when you receive a confirmation email by us.

Cancellation Policy #

You may cancel your registration for the exam up to 7 days before the scheduled date of the exam without the need for a reason. If you do not take the exam at the scheduled time without having cancelled at least 7 days ahead of time, you will be required to submit a medical certificate for the scheduled time of the exam, otherwise the central exam office will be notified about the missed exam by us. In the event that you cancel your exam registration and this is your first cancellation for other than medical reasons, you may contact us to schedule an alternative exam date.

In case you registered for an exam but fail to meet the requirements installed to be admissible for the exam to begin with, you may cancel your exam without consequences. However, please contact us if this is the case, so we can take a closer look at the situation.

Exam Location #

All exams will be held in room 2.300 of INF 205 (Mathematikon).

Course Material #

Lecture Notes #

• Lecture Notes (2023-04-17, §1-§3)
• Lecture Notes (2023-04-23, §1-§4.4) (Lemma 4.10 incorrect)
• Lecture Notes (2023-04-26, §1-§4.6)
• Lecture Notes (2023-05-07, §1-§5.3)
• Lecture Notes (2023-05-13, §1-§5.6)
• Lecture Notes (2023-05-21, §1-§5.8)
• Lecture Notes (2023-05-28, §1-§6)
• Lecture Notes (2023-06-06, §1-§9)
• Lecture Notes (2023-06-17, §1-§11.3)
• Lecture Notes (2023-06-23, §1-§13.1)
• Lecture Notes (2023-07-01, §1-§13.2)
• Lecture Notes (2023-07-09, §1-§14)
• Lecture Notes (2023-07-14, §1-§16)
• Lecture Notes (2023-07-18, §1-§16)

Exercises #

• Exercise sheet 01 – Solutions 01, Code 01
• Exercise sheet 02 – Solutions 02, Code 02
• Exercise sheet 03 – Solutions 03, Code 03
• Exercise sheet 04 – Solutions 04, Code 04
• Exercise sheet 05 – Solutions 05, Code 05
• Exercise sheet 06 – Solutions 06, Code 06
• Exercise sheet 07 – Solutions 07, Code 07
• Exercise sheet 08 – Solutions 08
• Exercise sheet 09 – Solutions 09
• Exercise sheet 10 – Solutions 10
• Exercise sheet 11 – Solutions 11
• Exercise sheet 12 – Solutions 12
• Exercise sheet 13 – Solutions 13

Literature #

This class is self sufficient and does not rely on outside literature. If you would like to read up additionally, we can however recommend the following literature:

• Nocedal, Wright: Numerical Optimization
• Geiger, Kanzow: Theorie und Numerik restringierter Optimierungsaufgaben
• Ulbrich, Ulbrich: Nichtlineare Optimierung