Optimization & Lagrange Multipliers

Study Sheet

Optimization & Lagrange Multipliers

Critical points, the second-derivative test, constrained maxima

Free and Constrained Optimization

Tip
Critical points and the D-test

Interior extrema need ∇f=0\nabla f = \mathbf 0. Classify with D=fxxfyy−fxy2D = f_{xx}f_{yy} - f_{xy}^2: D>0D > 0 with fxx>0f_{xx} > 0 is a MIN, D>0D > 0 with fxx<0f_{xx} < 0 a MAX, and D<0D < 0 a SADDLE — the genuinely new 2-variable phenomenon, downhill one way and uphill another.

Tip
Lagrange: gradients align at constrained extrema

To optimize ff subject to g=cg = c, solve ∇f=λ∇g\nabla f = \lambda\nabla g: at the best point the level curve of ff is TANGENT to the constraint, so the gradients are parallel. Maximizing xyxy on x+y=10x + y = 10 gives x=y=5x = y = 5 — AM–GM, rediscovered by calculus.

Example
A linear function on a circle

Maximize x+2yx + 2y on x2+y2=5x^2 + y^2 = 5: Lagrange forces (x,y)∥(1,2)(x,y) \parallel (1,2), giving (1,2)(1, 2) and the maximum 55. In general ax+byax + by on a radius-rr circle peaks at ra2+b2r\sqrt{a^2+b^2} — Cauchy–Schwarz with a geometric proof.

Side note
Significance: everything optimal is a critical point

Prices in economics, shapes of soap films, maximum-likelihood estimates in statistics, minimum-energy configurations in chemistry — all sit where a gradient vanishes, usually subject to constraints. Lagrange multipliers even carry meaning: λ\lambda is the "shadow price," how much the optimum improves per unit of loosened constraint.

Try it
Try it: a fenced field

Maximize the area xyxy of a rectangle with 2x+y=402x + y = 40 (a river covers one side). Work: Lagrange or substitute: A=x(40−2x)A = x(40 - 2x), A′=40−4x=0A' = 40 - 4x = 0, x=10x = 10, y=20y = 20, area 200200. Note the pattern: the constrained side gets half its "budget" — Lagrange conditions encode it automatically.

Proofs & Why It Matters

Tip
Proof sketch: the second-derivative D-test

Near a critical point, Taylor gives f≈f(a,b)+12(fxxh2+2fxyhk+fyyk2)f \approx f(a,b) + \tfrac12\left(f_{xx}h^2 + 2f_{xy}hk + f_{yy}k^2\right) — a quadratic form Q(h,k)Q(h,k). Complete the square: Q=fxx(h+fxyfxxk)2+Dfxxk2Q = f_{xx}\left(h + \tfrac{f_{xy}}{f_{xx}}k\right)^2 + \tfrac{D}{f_{xx}}k^2 with D=fxxfyy−fxy2D = f_{xx}f_{yy} - f_{xy}^2.

If D>0D > 0 both terms share the sign of fxxf_{xx} (a definite bowl: min or max); if D<0D < 0 the two terms disagree, so QQ takes both signs — a saddle. ■\blacksquare

Tip
Proof: why ∇f = λ∇g at a constrained extremum

Let r(t)\mathbf r(t) be any curve inside the constraint surface g=cg = c through the extremum at t=0t = 0. Then t↦f(r(t))t \mapsto f(\mathbf r(t)) has an extremum at 00, so 0=ddtf(r(t))∣0=∇f⋅r′(0)0 = \tfrac{d}{dt}f(\mathbf r(t))\big|_0 = \nabla f\cdot\mathbf r'(0).

Thus ∇f\nabla f is orthogonal to EVERY tangent direction of the constraint — but the vectors orthogonal to that whole tangent plane are exactly the multiples of ∇g\nabla g. Hence ∇f=λ∇g\nabla f = \lambda\nabla g. ■\blacksquare

Going Deeper: Explanations & Worked Problems

Concept
Why critical points, and what can go wrong

At an interior maximum or minimum, every slice through the point is a one-variable function with an extremum there, so every directional derivative vanishes — hence ∇f=0\nabla f = \mathbf 0.

But the converse fails in a way one variable never shows: at a SADDLE, the surface rises along one line and falls along another, with ∇f=0\nabla f = \mathbf 0 all the same (f=x2−y2f = x^2 - y^2 at the origin is the model: uphill along the xx-axis, downhill along the yy-axis). That is why the DD-test exists: near a critical point, ff is governed by its quadratic part 12(fxxh2+2fxyhk+fyyk2)\tfrac12(f_{xx}h^2 + 2f_{xy}hk + f_{yy}k^2), and D=fxxfyy−fxy2D = f_{xx}f_{yy} - f_{xy}^2 decides whether that quadratic keeps one sign (a genuine bowl or dome) or changes sign (a saddle). And remember the boundary: on a closed region, extrema can sit on the edge with ∇f≠0\nabla f \ne \mathbf 0 — that is precisely the situation Lagrange multipliers handle.

Example
Worked: classify every critical point of f = x³ − 3x + y²

Step 1 — set the gradient to zero: fx=3x2−3=0f_x = 3x^2 - 3 = 0 gives x=±1x = \pm1; fy=2y=0f_y = 2y = 0 gives y=0y = 0. Critical points: (1,0)(1, 0) and (−1,0)(-1, 0).

Step 2 — second partials: fxx=6xf_{xx} = 6x, fyy=2f_{yy} = 2, fxy=0f_{xy} = 0, so D=12xD = 12x.

Step 3 — test (1,0)(1, 0): D=12>0D = 12 > 0 and fxx=6>0f_{xx} = 6 > 0 — both slice curvatures upward: a LOCAL MIN, value f(1,0)=1−3=−2f(1,0) = 1 - 3 = -2.

Step 4 — test (−1,0)(-1, 0): D=−12<0D = -12 < 0 — a SADDLE, no extremum, even though the gradient vanishes.

Step 5 — global view: as x→−∞x \to -\infty, x3x^3 drags f→−∞f \to -\infty, so the local min at (1,0)(1,0) is NOT global — a reminder that the DD-test speaks only about a neighborhood.

Example
Worked: Lagrange multipliers on x + 2y with x² + y² = 5, no steps skipped

Step 1 — set up ∇f=λ∇g\nabla f = \lambda\nabla g: f=x+2yf = x + 2y, g=x2+y2g = x^2 + y^2, so (1,2)=λ(2x,2y)(1, 2) = \lambda(2x, 2y), giving 1=2λx1 = 2\lambda x and 2=2λy2 = 2\lambda y.

Step 2 — eliminate λ\lambda: divide the equations: 12=xy\tfrac12 = \tfrac xy, so y=2xy = 2x.

Step 3 — enforce the constraint: x2+(2x)2=5x^2 + (2x)^2 = 5 gives 5x2=55x^2 = 5, x=±1x = \pm1, hence the candidates (1,2)(1, 2) and (−1,−2)(-1, -2).

Step 4 — evaluate ff at each: f(1,2)=1+4=5f(1,2) = 1 + 4 = 5 and f(−1,−2)=−5f(-1,-2) = -5. The constraint circle is closed and bounded, so these are the global max and min: maximum 55.

Step 5 — see it geometrically: the level lines x+2y=cx + 2y = c are parallel lines sliding outward as cc grows; the largest cc still touching the circle touches it TANGENTLY, at the point where the radius is parallel to (1,2)(1,2) — which is exactly what Step 1 said.