auto-saved โœ“

Deep Learning โ€” Week 5 (Hinglish Notes)

Adaptive learning rate: AdaGrad, RMSProp, AdaDelta, Adam, AdaMax/MaxProp, NAdam + LR schedulers (Cyclical, Cosine annealing, Warm-start)  ยท  Prof. Mitesh Khapra ke slides (Module 5.9โ€“5.10) + lecture transcripts + PYQs + PA & GA Week 5

Page kaise use karein Upar toolbar: Pen / Highlight / Line / Text / Eraser, colour aur size choose karo. Jo bhi likhoge woh browser me auto-save hota hai (localStorage). Dusre laptop pe le jaana ho ya backup chahiye to โฌ‡ Save file / โฌ† Load use karo. ๐Ÿ™ˆ Practice dabao to saare answers blur ho jayenge; answer pe click karo to dikh jayega. Shortcuts: P pen ยท H highlight ยท L line ยท T text ยท E eraser ยท Esc read ยท Ctrl+Z / Ctrl+Y.

Contents

  1. Poori kahani ek nazar me
  2. Exam me kya aata hai
  3. Problem: sparse features
  4. AdaGrad
  5. RMSProp
  6. AdaDelta
  7. Adam
  8. Bias correction kyu?
  9. Lโˆž norm: MaxProp & AdaMax
  10. NAdam
  11. LR schedulers
  12. Comparison table
  13. Formula sheet
  14. PYQs solved
  15. Practice Assignment (PA)
  16. Graded Assignment (GA)
  17. My notes

1. Poori kahani ek nazar me (professor actually kya bol rahe hain)

Week 4 me humne dekha: GD, Momentum, NAG โ€” sab me ek hi learning rate $\eta$ sab weights ke liye tha. Week 5 ka pura topic ek sawaal ka jawab hai:

Main sawaal "Kya har parameter ka apna alag learning rate ho sakta hai, jo uski gradient history dekh ke khud adjust ho?" โ€” isi ko adaptive learning rate bolte hain.

Professor ek chain me har algorithm ko pichhle wale ki kamzori fix karte hue introduce karte hain. Yeh chain yaad rakho, aadha exam isi se nikal jaata hai:

AlgorithmKya problem fix kiyaNayi problem
AdaGradSparse feature wale weight ko bhi bada update mile$v_t$ hamesha badhta hai โ†’ LR itna gir jaata hai ki dense param (jaise $b$) ruk jaata hai
RMSProp$v_t$ ko decay karo (moving average) โ†’ LR zero tak nahi girtaAbhi bhi initial $\eta_0$ choose karna padta hai; oscillation aa sakta hai
AdaDelta$\eta_0$ hi hata do! Numerator me bhi history ($u_{t-1}$)โ€”
AdamRMSProp ka denominator + momentum ($m_t$) + bias correctionKabhi kabhi generalize kam karta hai
AdaMax / MaxProp$L_2$ ki jagah $L_\infty$ (max) norm โ†’ zero gradient pe LR nahi badhtaโ€”
NAdamAdam + NAG ka look-aheadโ€”

Last me Module 5.10: learning rate schedules โ€” LR ko time ke saath kaise badlein (cyclical, cosine annealing with warm restart, warm-start). Yahan main idea: saddle point se nikalne ke liye LR ko kabhi kabhi badhana bhi padta hai.

2. Exam me kya aata hai padh ke shuru karo

TopicKitni baar (PYQ)Type
AdaGrad step-by-step numerical ($w_1, v_1, w_2$)Sep'24, Jan'25, ET Jan'26 + PA + GASA comprehension (3โ€“5 marks)
"Kaun sa optimizer adaptive LR / momentum use karta hai?"Sep'24, T2'25, Sep'25, ET Sep'25MSQ
Adam me bias correction kyuJan'25, Sep'25, PAMCQ (same question repeat!)
Adam = RMSProp + momentum; NAdam = NAG + AdamSep'24, Jan'26MCQ
Algorithm โ†” property matchingJan'26MCQ 4 marks
Moving average / step decay / line search numericalsJan'25, ET Sep'25, Jan'26SA/MCQ
Cosine annealing value, RMSProp ฮฒ effect, AdaMax factsPASA/MSQ
โœ” StrategyAdaGrad ka loop haath se 2 iteration chalana practice karo, Adam ka formula + bias correction ka reason ratt lo, aur section 12 ka comparison table. Itne me ~80% Week 5 cover.

3. Problem: sparse features Module 5.9

Ek sigmoid neuron lo, inputs $x^1,x^2,x^3,x^4$, weights $w_1..w_4$:

$$\nabla w_i = (f(x)-y)\,f(x)\,(1-f(x))\cdot x^i$$

Dekho โ€” gradient me $x^i$ multiply ho raha hai. Agar feature $x^2$ sparse hai (zyadatar inputs me $x^2=0$), to $\nabla w_2 = 0$ zyadatar time. Matlab $w_2$ ko update hi nahi milta.

Professor ka point Agar $x^2$ sparse hai aur important bhi hai (jaise rare word jo sentiment decide karta hai), to jab bhi uska gradient aaye, hume us update ko zyada seriously lena chahiye โ†’ us parameter ka LR bada hona chahiye.

Toy experiment (slides): sirf 2 params $w, b$. $b$ ka input hamesha 1 hai (dense, sparse nahi kar sakte), to $x$ ko sparse banaya: 500 points, ~80% me $x=0$.

Result: GD, Momentum, NAG โ€” teeno pehle sirf vertical ($b$) direction me chalte hain, $w$ direction me bahut kam. Kyunki $b$ ko har step pe update milta hai, $w$ ko kabhi kabhi.

โš  Contour se sparse/dense pehchanna (GA Q1 type) Sparse feature โ†’ us weight ka gradient chhota โ†’ loss us direction me dheere badalta hai โ†’ contour us axis ke along lamba (stretched) hota hai.
Ellipse $w_2$ axis ke along lamba (vertical) โ†’ $x_2$ sparse, $x_1$ dense.
Ellipse $w_1$ axis ke along lamba (horizontal) โ†’ $x_1$ sparse, $x_2$ dense.
๐Ÿ“ Exam me aise poocha โ€” abhi try karo
Contour ellipse $w_2$ axis ke along lamba hai. Kaunsa feature sparse? GA W5 Q1
$x_2$ sparse ($w_2$ direction me loss flat), $x_1$ dense.
Tilted ellipse, GD $(-6,0)$ se start. Kya path seedha horizontal hoga? GA W5 Q2
Nahi (False) โ€” GD contour ke perpendicular chalta hai, tilted ellipse pe gradient ka $w_2$ component bhi hai.

4. AdaGrad (Adaptive Gradient) Module 5.9 numericals aate hain

IntuitionJis parameter ko jitne zyada updates mil chuke hain, uska LR utna zyada decay karo. Kam updates mile (sparse) โ†’ LR bada rahe.
AdaGrad update $$v_t = v_{t-1} + (\nabla w_t)^2$$ $$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t+\epsilon}}\,\nabla w_t$$ Effective learning rate: $\displaystyle \eta_t = \frac{\eta_0}{\sqrt{v_t+\epsilon}}$   (har parameter ka apna $v_t$)

Unroll karo:

$$v_t = (\nabla w_0)^2 + (\nabla w_1)^2 + \dots + (\nabla w_t)^2$$

Yaani saare past gradients ke squares ka sum โ€” koi decay nahi. Isliye:

โš  AdaGrad ki 3 important properties (MCQ favourite)
  1. $v_t$ kabhi decrease nahi hota (sirf squares add ho rahe hain, sab $\ge 0$). Gradient 0 ho jaaye to $v_t$ constant rehta hai, kam nahi hota.
  2. Isliye effective LR monotonically decreasing (ya constant) hai. Kabhi badh nahi sakta.
  3. Flipside: dense param ka LR itna gir jaata hai ki woh update hona band ho jaata hai โ€” minimum ke paas AdaGrad slow ho jaata hai.
Bonus: slides kehte hain square root hata do to practice me kaam achha nahi karta.
โœ” Worked example (PA Q1) $\nabla w = 3, 4, 12$; $v_0=0$, $\eta=1$, $\epsilon=0$.
$v_1=9$, $v_2=9+16=25$, $v_3=25+144=169$ โ†’ effective LR $=1/\sqrt{169}=1/13\approx\mathbf{0.077}$.
Agar aage gradient 0 โ†’ $v$ = 169 hi rahega, LR 1/13 pe atak jayega, aur $w$ hilega nahi (update = LR ร— 0).
โš  Gradient-profile graph question (GA Q3) Gradient chahe upar-neeche jaye, AdaGrad ka effective LR graph hamesha neeche jaata hua (decreasing) curve hota hai. Badhta hua curve = $v_t$ ka graph (trap option). Gradient jaisa shape = galat.
๐Ÿ“ Exam me aise poocha โ€” abhi try karo
$L=w^2$, AdaGrad, $\eta=1$, $w_0=2$, $v_{-1}=0$. $w_1$, $v_1$, $w_2$? Quiz-2 Sep'24
$g_0=4,v_0=16,w_1=1$; $g_1=2,v_1=20$; $w_2=1-2/\sqrt{20}=0.553$.
$L=w^3+2w$, $\eta=0.5$, $w_0=2$. $w_1$ aur $v_1$? Quiz-2 Jan'25
$g_0=14,v_0=196,w_1=1.5$; $g_1=8.75$, $v_1=272.56$; $w_2=1.235$.
Gradients 3, 4, 12; $\eta=1$. Iteration 3 pe effective LR? Phir sab gradients 0 ho jaayein to? PA Q1
$v_3=169$ โ†’ $1/13=0.077$. Gradient 0 โ†’ $v$ = 169 constant, $w$ ruk jaata hai.
Gradient upar-neeche ho raha hai โ€” AdaGrad ka effective LR ka graph kaisa? $\eta_{100}>\eta_6$? GA Q3, Q5
Hamesha girta hua curve. $\eta_{100}>\eta_6$ โ†’ False.
$f=3(x^2+y^2)$: GD 2 steps ($\eta=0.1$) ke baad $(x_2,y_2)=\alpha(x_0,y_0)$; AdaGrad ke liye aisa $\alpha$? ET Jan'26
GD: $\alpha=0.4^2=0.16$. AdaGrad: pehla step $0.1\cdot\text{sign}$ har coordinate me โ†’ common $\alpha$ nahi โ†’ 0.

5. RMSProp Module 5.9 most asked

IntuitionAdaGrad denominator ko bahut aggressively badhata hai. To denominator ko bhi decay karo โ€” saare history ka sum nahi, exponentially weighted moving average lo.
RMSProp update $$v_t = \beta\, v_{t-1} + (1-\beta)(\nabla w_t)^2$$ $$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t+\epsilon}}\,\nabla w_t$$

Unroll ($\beta=0.9$, $v_{-1}=0$):

$$v_0 = 0.1\,\nabla b_0^2,\quad v_1 = 0.09\,\nabla b_0^2 + 0.1\,\nabla b_1^2,\quad v_2 = 0.081\,\nabla b_0^2 + 0.09\,\nabla b_1^2 + 0.1\,\nabla b_2^2$$

$$v_t = (1-\beta)\sum_{\tau=0}^{t}\beta^{\,t-\tau}\,(\nabla w_\tau)^2$$

Purane gradients ka weight $\beta^{t-\tau}$ se chhota hota jaata hai โ†’ woh "bhool" jaate hain.

โœ” AdaGrad vs RMSProp โ€” yeh line ratt lo AdaGrad: LR monotonically decreases (ever-growing denominator).
RMSProp: LR increase, decrease ya constant โ€” kuch bhi ho sakta hai, kyunki denominator moving average hai. Flat region (chhota gradient) aaya to $v_t$ girta hai โ†’ LR wapas badh jaata hai.

ฮฒ ka effect (PA Q3)

Constant gradient $g$ ke liye: $v_t=(1-\beta^{t+1})g^2$, to effective LR $=\dfrac{\eta}{|g|\sqrt{1-\beta^{t+1}}}$.

Bada ฮฒ (0.95) โ†’ $\beta^{t+1}$ bada โ†’ $v_t$ chhota โ†’ LR bada โ†’ LR dheere decay karta hai. Chhota ฮฒ (0.2) โ†’ jaldi decay.

Oscillation problem

Slide: $\eta=0.1$, $\beta=0.5$ pe RMSProp AdaGrad se jaldi converge hua, lekin minimum ke paas oscillate karne laga. Kyun? Gradients chhote hone pe $v_t$ bhi chhota โ†’ LR ~constant reh jaata hai โ†’ minimum ke aas paas jhoolta rehta hai. Fix: initial $\eta$ sahi choose karo ($\eta=0.05$ se theek ho gaya).

ฮทโ‚€ ka dard

Steep region me $v_t=1.25$, flat me $v_t=0.1$ maan lo (slide ke numbers, $\sqrt{}$ ignore karke):

steep ($v_t=1.25$)flat ($v_t=0.1$)
$\eta_0=0.6$0.486
$\eta_0=0.1$0.08 โœ” (steep me chhota achha)1

Steep me 0.08 achha, flat me 6 achha โ€” koi ek $\eta_0$ dono jagah best nahi. Isliye hum chahte hain ki numerator bhi gradient ke saath badle โ†’ yahi AdaDelta ka motivation hai.

โš  Drawbacks (AdaGrad + RMSProp dono) Initial LR, initial params aur initial gradients ke prati sensitive. AdaGrad me shuru ke gradients bade the to baaki training LR chhota hi rahega, aur baad me gentle region aaye to LR badhane ka koi tareeka nahi.
๐Ÿ“ Exam me aise poocha โ€” abhi try karo
$v_t=0.9v_{t-1}+0.1\nabla w_t$, gradients 5, 3, 4, $v_{-1}=0$. $v_2$? Quiz-2 Jan'25
0.5 โ†’ 0.75 โ†’ 1.075.
AdaGrad ke sum ko moving average se replace karne ka asar? PA Q2
LR zero tak decay nahi hota, low-gradient region me wapas badh sakta hai.
ฮฒ โˆˆ {0.2, 0.6, 0.95} โ€” kiska LR sabse dheere decay? PA Q3
ฮฒ = 0.95 (bada ฮฒ โ†’ chhota $v_t$ โ†’ bada LR).
Gradients [1, 0.9, 0.6, 0.01, 0.1, 0.2, 0.5โ€ฆ], ฮฒ=0.9, ฮท=0.1. RMSProp $\eta_6$? GA Q6
$v_6=0.154$ โ†’ $0.1/\sqrt{0.154}=0.25$.
"Exponentially decaying average of squared gradients to prevent LR from vanishing" โ€” kaun? Quiz-2 Jan'26 (matching)
RMSprop. ("Total sum of all past squared gradients" = AdaGrad.)

6. AdaDelta Module 5.9

IntuitionInitial learning rate $\eta_0$ set hi nahi karna. Numerator me constant $\eta$ ki jagah past updates ($\Delta w$) ka moving average daal do.
AdaDelta (har iteration t) 1. $\nabla w_t$ compute karo
2. $v_t = \beta v_{t-1} + (1-\beta)(\nabla w_t)^2$
3. $\displaystyle \Delta w_t = -\frac{\sqrt{u_{t-1}+\epsilon}}{\sqrt{v_t+\epsilon}}\,\nabla w_t$
4. $w_{t+1} = w_t + \Delta w_t$
5. $u_t = \beta u_{t-1} + (1-\beta)(\Delta w_t)^2$

Slide experiment: RMSProp ko $\eta_0=0.013$ diya, AdaDelta ko kuch nahi, dono $w_0=b_0=-4$ se โ€” AdaDelta jaldi converge hua kyunki uska LR "wisely adapt" hota hai.

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
"Automatically adjusts updates based on units of parameters, no manual learning rate" โ€” kaun? Quiz-2 Jan'26 (matching)
AdaDelta.
Kaun current gradient + gradient history se LR adapt karta hai: exp-decay GD, RMSProp, NAG+step decay, ADAM, Ada-Delta? Quiz-2 Sep'24 MSQ
RMSProp, ADAM, Ada-Delta.

7. Adam (Adaptive Moments) Module 5.9 most asked

IntuitionRMSProp/AdaDelta wala denominator (LR decay control) + gradients ki cumulative history = momentum numerator me. Dono moments: $m_t$ (1st moment, mean) aur $v_t$ (2nd moment, uncentered variance).
Adam update $$m_t = \beta_1 m_{t-1} + (1-\beta_1)\nabla w_t \qquad \hat m_t = \frac{m_t}{1-\beta_1^{\,t}}$$ $$v_t = \beta_2 v_{t-1} + (1-\beta_2)(\nabla w_t)^2 \qquad \hat v_t = \frac{v_t}{1-\beta_2^{\,t}}$$ $$w_{t+1} = w_t - \frac{\eta}{\sqrt{\hat v_t}+\epsilon}\,\hat m_t$$ Typical: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon=10^{-8}$
โœ” Shortcut: Adam ka pehla step $t=1$: $m_1=(1-\beta_1)g$, $\hat m_1 = g$; $v_1=(1-\beta_2)g^2$, $\hat v_1=g^2$. To step $=\eta\,g/|g| = \eta\cdot\text{sign}(g)$.
Adam ka pehla step hamesha size $\eta$ ka hota hai, gradient kitna bhi bada/chhota ho. (Isliye GA Q7 me Adam $-2\to-1.9\to-1.8\ldots$ โ€” har step lagbhag 0.1.)

"Which optimizer to use?" (professor ki rai):

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
Kaunsa algorithm RMSprop aur momentum ke fayde combine karta hai? Quiz-2 Sep'24
Adam.
Kaun momentum use karte hain: NAG, AdaGrad, RMSProp, AdaDelta, ADAM? ET Sep'25 MSQ
NAG, ADAM. (RMSProp/AdaDelta ka average $g^2$ ka hai โ€” momentum nahi.)
$L\approx w^2$, $w_0=-2$, $\eta=0.1$, 10 steps: Adam vs Momentum GD? GA Q7
MGD minimum cross karta hai aur 10 steps ke baad ~0 pe hai; Adam ~0.1/step chalta hai (โˆ’1.02), cross nahi karta. True: MGD crosses, Adam doesn't, MGD closer.
Adaptive LR kaun use karte hain: ADAM, RMSprop, Momentum GD, NAG, SGD, Adagrad? Quiz-2 T2'25 / Sep'25 MSQ
ADAM, RMSprop, Adagrad. Momentum/NAG/SGD nahi.

8. Bias correction kyu chahiye? theory repeat MCQ

Hum $m_t$ isliye le rahe hain kyunki hume $E[\nabla w]$ (gradient ka expected value) chahiye, ek point estimate nahi. Ideally $E[m_t] = E[\nabla w_t]$ hona chahiye. Check karo:

$$m_t = (1-\beta)\sum_{\tau=1}^{t}\beta^{t-\tau}\nabla w_\tau$$

Assume saare $\nabla w_\tau$ same distribution se ($E[\nabla w_\tau]=E[\nabla w]$):

$$E[m_t] = E[\nabla w](1-\beta)(\beta^{t-1}+\dots+\beta^0) = E[\nabla w](1-\beta)\frac{1-\beta^t}{1-\beta} = E[\nabla w]\,(1-\beta^t)$$

To $m_t$ biased hai โ€” zero ki taraf (factor $1-\beta^t$ jo shuru me chhota hai). Fix: $\hat m_t = \dfrac{m_t}{1-\beta^t}$ โ†’ $E[\hat m_t]=E[\nabla w]$. โœ”

Ek line ka answerBias correction isliye kyunki $m_0=v_0=0$ se start karne pe shuru ke iterations me moving averages underestimated (zero ki taraf biased) hote hain. Correction se unka expected value sahi ho jaata hai.

Bias correction na karein to kya hota hai? (slide numerics)

$\beta_2=0.999$, $\nabla w_0=0.1$, $\eta=1$:

Bina correctionCorrection ke saath
$t=0$$v_0 = 0.001(0.01)=10^{-5}$ โ†’ LR $=1/\sqrt{10^{-5}}=316.2$$\hat v_0 = 10^{-5}/0.001=0.01$ โ†’ LR $=1/0.1=10$
agla step, $\nabla w=0$$v=0.0000099$ โ†’ LR $\approx316.4$$\hat v\approx0.0052$ โ†’ LR $\approx13.8$

Paper ka quote: "a lack of initialization bias correction would lead to initial steps that are much larger". Yaani bina correction ke shuru ke steps bahut bade โ€” correction unhe attenuate (kam) karta hai.

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
Why is bias correction needed in Adam? Quiz-2 Jan'25 ยท Sep'25 ยท PA
Shuru me first aur second moments ke underestimated moving averages correct karne ke liye.
"Includes bias-correction terms to account for initialization at the origin" โ€” kaun? Quiz-2 Jan'26 (matching)
Adam.

9. $L_\infty$ norm: MaxProp aur AdaMax Module 5.9

$L_p$ norm: $\|x\|_p = (|x_1|^p+\dots+|x_n|^p)^{1/p}$. Jaise $p\to\infty$, yeh $\max(|x_1|,\dots,|x_n|)$ ban jaata hai. Bade $p$ pe $|x|^p$ numerically unstable hota hai, to seedha max le lo.

Adam ka $v_t$ effectively $L_2$ norm hai. $L_\infty$ use karo:

Max-norm denominator $$v_t = \max(\beta_2 v_{t-1},\ |\nabla w_t|) \quad\text{(dhyan do: } |\nabla w_t|, \text{ square nahi)}$$ MaxProp (RMSProp ka max version): $\displaystyle w_{t+1}=w_t-\frac{\eta}{v_t+\epsilon}\nabla w_t$
AdaMax (Adam ka max version): $\displaystyle w_{t+1}=w_t-\frac{\eta}{v_t+\epsilon}\hat m_t$   ($\hat m_t$ bias-corrected, $v_t$ ka bias correction NAHI)
โš  Do key facts
  1. Max norm ko bias correction nahi chahiye โ€” woh zero-initialization ki taraf biased nahi hota (max turant asli gradient pakad leta hai).
  2. Zero gradient pe LR nahi badhta (lagbhag). Exponential averaging (bias correction ke saath bhi) zero gradient pe LR badha deta hai.
Max norm ko AdaMax naam se usi paper me propose kiya gaya tha jisme Adam aaya tha.
โœ” Slide example (50% inputs zero, $\beta_2=0.999$) Max norm: $v_0=\max(0,1)=1$ โ†’ LR 1; $v_1=\max(0.999,0)=0.999$ โ†’ LR 1.001; $v_2=\max(0.999\cdot0.999, 1)=1$ โ†’ LR 1. Almost constant.
Exp. avg + BC: $v_0=0.001$, $\hat v_0=1$ โ†’ LR 1; $v_1=0.000999$, $\hat v_1=0.000999/(1-0.999^2)\approx0.4997$ โ†’ LR $1/\sqrt{0.5}\approx1.41$. Zero gradient pe bhi LR badh gaya!

Smooth convex surface pe exponential averaging bhi theek hai (LR badhne se nuksaan nahi), dono converge karte hain. General rule: steep surface pe chhota LR, gentle pe bada LR chahiye.

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
AdaMax: $v_t=\max(\beta_2v_{t-1},|\nabla w_t|)$? $v_t$ pe bias correction? Zero gradients pe LR tezi se badhta hai? PA Q5 (MSQ)
Pehle do sahi (BC sirf $m_t$ pe). Teesra galat โ€” max norm LR nahi badhata.

10. NAdam (Nesterov + Adam) Module 5.9

NAG momentum se better tha, to Adam me NAG daal do โ€” bas $m_t$ modify karna hai.

Problem: normal NAG me $m_{t-1}$ do baar use hota hai (look-ahead gradient me aur momentum me) โ†’ do weight updates, bade network ke liye costly.

Trick (rewritten NAG): look-ahead pichhle step me hi kar lo:

$$g_{t+1}=\nabla w_t,\quad m_{t+1}=\beta m_t + \eta g_{t+1},\quad w_{t+1}=w_t-\eta(\beta m_{t+1}+g_{t+1})$$

NAdam update $$m_{t+1}=\beta_1 m_t+(1-\beta_1)\nabla w_t,\qquad \hat m_{t+1}=\frac{m_{t+1}}{1-\beta_1^{t+1}}$$ $$v_{t+1}=\beta_2 v_t+(1-\beta_2)(\nabla w_t)^2,\qquad \hat v_{t+1}=\frac{v_{t+1}}{1-\beta_2^{t+1}}$$ $$w_{t+1}=w_t-\frac{\eta}{\sqrt{\hat v_{t+1}}+\epsilon}\left(\beta_1\hat m_{t+1}+\frac{(1-\beta_1)\nabla w_t}{1-\beta_1^{t+1}}\right)$$

Matlab Adam ke $\hat m$ ki jagah: $\beta_1\hat m_{t+1}$ (look-ahead momentum) + current gradient ka bias-corrected hissa.

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
Kaunsa algorithm NAG ko adaptive moment estimation ke saath combine karta hai? Quiz-2 Jan'26
NAdam.
"NAdam incorporates a Nesterov-style look-ahead into Adam's first-moment term" โ€” True? PA Q5
True.

11. Learning-rate schedulers Module 5.10 cosine numerical aata hai

Epochs basedValidation basedGradient based (adaptive)
Step decay, Exponential decay, Cyclical, Cosine annealing, Warm-restartLine search, Log searchAdaGrad, RMSProp, AdaDelta, Adam, AdaMax, NAdam, AMSGrad, AdamW

Cyclical LR (CLR) โ€” kyu?

Loss surface pe saddle point hai. Agar LR exponentially decay ho raha hai, saddle ke paas pahunchte pahunchte LR itna chhota ki bahar nikal hi nahi sakte. Rationale: loss minimize karne me mushkil zyadatar saddle points se aati hai, bure local minima se nahi. To LR ko cyclically badhao-ghatao โ€” adaptive methods bhi help karte hain par unka extra computational cost hai, CLR simple alternative hai.

Triangular CLR $$\eta_t=\eta_{min}+(\eta_{max}-\eta_{min})\cdot\max\!\left(0,\ 1-\left|\frac{t}{\mu}-2\left\lfloor1+\frac{t}{2\mu}\right\rfloor+1\right|\right)$$ $\mu$ = step size (aadha cycle). Example: $t=20,\mu=20$: $\lfloor1+20/40\rfloor=1$, $|1-2+1|=0$, $\max(0,1)=1$ โ†’ $\eta_t=\eta_{max}$. Yaani $t=\mu$ pe peak, $t=0, 2\mu$ pe minimum.

Cosine annealing (warm restart)

Cosine annealing $$\eta_t=\eta_{min}+\frac{\eta_{max}-\eta_{min}}{2}\left(1+\cos\left(\pi\,\frac{t\,\%\,(T+1)}{T}\right)\right)$$ $t=0$: $\cos 0=1$ โ†’ $\eta_{max}$.   $t=T$: $\cos\pi=-1$ โ†’ $\eta_{min}$.   $t=T+1$: $\%$ se wapas 0 โ†’ achanak $\eta_{max}$ (isliye "warm restart").
$T$ = restart interval.
โœ” PA Q6$\eta_{min}=0,\eta_{max}=1,T=10,t=5$: $5\%11=5$, $\cos(5\pi/10)=\cos(\pi/2)=0$ โ†’ $\eta_5=\frac12(1+0)=\mathbf{0.5}$.
Shortcut: $t=T/2$ pe hamesha beech ka value $\frac{\eta_{max}+\eta_{min}}{2}$.
โš  Cosine vs exponential decay ka fayda (PA Q7)Cosine annealing periodically LR ko wapas $\eta_{max}$ pe restart karta hai โ†’ optimizer saddle points se escape kar sakta hai. Monotone decay me yeh possible nahi.

Slide example: cosine annealing ne 46th iteration pe minimum pakda (aur chalao to cross kar jaata hai โ€” early stopping se roll back kar sakte ho).

Warm-start (warmup)

Normally LR high se start karke decay karte hain. Ulta: shuru me chhota LR rakho aur dheere badhao โ€” model "warm up" hota hai aur better converge karta hai. Transformers me bahut use hota hai (e.g. warmupSteps=4000).

๐Ÿ“ Exam me aise poocha โ€” abhi try karo
Cosine: $\eta_{min}=0,\eta_{max}=1,T=10$. $\eta_5$? PA Q6
$\cos(\pi/2)=0$ โ†’ 0.5.
Saddle points pe cosine annealing ka exponential decay se fayda? PA Q7
Periodic restart to $\eta_{max}$ โ†’ saddle se escape.
$\eta=0.1$, 100 updates/epoch, har 150 updates pe LR half. 2nd epoch ke end pe? ET Sep'25
200 updates โ†’ 1 halving โ†’ 0.05.
Backtracking: $J(w_{t+1})\le J(w_t)-10\eta$. ฮท=1: +5, 0.8: โˆ’4, 0.4: โˆ’6, 0.1: โˆ’0.5. Sabse bada valid ฮท? Quiz-2 Jan'26
0.4.

12. Comparison table revision

AlgoDenominator ($v_t$)NumeratorBias corr.LR behaviour
AdaGrad$v_{t-1}+g^2$ (sum)$\eta\,g$โ€”Sirf ghatta hai
RMSProp$\beta v+(1-\beta)g^2$$\eta\,g$โ€”Ghat/badh/constant
AdaDelta$\beta v+(1-\beta)g^2$$\sqrt{u_{t-1}+\epsilon}\;g$โ€”No $\eta_0$ needed
Adam$\beta_2 v+(1-\beta_2)g^2$$\eta\,\hat m_t$$m$ aur $v$ donoMomentum + adaptive
MaxProp$\max(\beta v,|g|)$$\eta\,g$โ€”Zero grad pe stable
AdaMax$\max(\beta_2 v,|g|)$$\eta\,\hat m_t$sirf $m$Zero grad pe stable
NAdamAdam jaisa$\eta(\beta_1\hat m+\ldots g)$donoLook-ahead momentum

Note: AdaGrad/RMSProp/MaxProp/AdaDelta me $\sqrt{v_t+\epsilon}$ (sqrt ke andar $\epsilon$); Adam me $\sqrt{\hat v_t}+\epsilon$. MaxProp/AdaMax me sqrt nahi.

13. One-page formula sheet

AdaGrad$v_t=v_{t-1}+g_t^2,\ \ w\mathrel{-}=\frac{\eta}{\sqrt{v_t+\epsilon}}g_t$
RMSProp$v_t=\beta v_{t-1}+(1-\beta)g_t^2,\ \ w\mathrel{-}=\frac{\eta}{\sqrt{v_t+\epsilon}}g_t$
RMSProp unrolled$v_t=(1-\beta)\sum_{\tau=0}^t\beta^{t-\tau}g_\tau^2$; constant $g$: $v_t=(1-\beta^{t+1})g^2$
AdaDelta$\Delta w_t=-\frac{\sqrt{u_{t-1}+\epsilon}}{\sqrt{v_t+\epsilon}}g_t,\ \ u_t=\beta u_{t-1}+(1-\beta)\Delta w_t^2$
Adam$\hat m_t=\frac{m_t}{1-\beta_1^t},\ \hat v_t=\frac{v_t}{1-\beta_2^t},\ w\mathrel{-}=\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\epsilon}$
Bias$E[m_t]=(1-\beta^t)E[g]$
AdaMax$v_t=\max(\beta_2v_{t-1},|g_t|),\ w\mathrel{-}=\frac{\eta}{v_t+\epsilon}\hat m_t$
Cosine$\eta_t=\eta_{min}+\frac{\eta_{max}-\eta_{min}}{2}\left(1+\cos\frac{\pi(t\%(T+1))}{T}\right)$
Triangular CLR$\eta_t=\eta_{min}+(\eta_{max}-\eta_{min})\max(0,1-|t/\mu-2\lfloor1+t/2\mu\rfloor+1|)$

14. PYQs โ€” fully solved

Yeh saare sawaal upar theory ke beech me ๐Ÿ“ boxes me already aa chuke hain. Yahan poore options + full solution ke saath hain. Official answer keys hi likhe hain. Practice ke liye upar ๐Ÿ™ˆ Practice dabao.

A. AdaGrad numericals har quiz me

AdaGrad numerical ka recipe (3 step loop) 1. $g_t=\nabla L(w_t)$ nikalo   2. $v_t=v_{t-1}+g_t^2$   3. $w_{t+1}=w_t-\frac{\eta}{\sqrt{v_t}}g_t$
Dhyan: $v_t$ me current gradient pehle add karo, phir update karo. Aur question me "$w_1$ when t=1" ka matlab = ek update ke baad wala $w$.
Quiz-2 Sep'24Comprehension Q70โ€“72
$L(w)=w^2$, AdaGrad, $\eta=1$, $w_0=2$, $v_{-1}=0$, $\epsilon=0$. (i) $w_1$? (ii) $v_1$? (iii) $w_2$?
$\nabla L=2w$.
$t=0$: $g_0=4$, $v_0=16$, $w_1=2-\frac{1}{4}\cdot4=\mathbf{1}$.
$t=1$: $g_1=2$, $v_1=16+4=\mathbf{20}$, $w_2=1-\frac{2}{\sqrt{20}}=1-0.447=\mathbf{0.553}$ (range 0.549โ€“0.556).
Quiz-2 Jan'25Comprehension Q78โ€“81
$L(w)=w^3+2w$, AdaGrad, $\eta=0.5$, $w_0=2$, $v_{-1}=0$, $\epsilon=0$. Find $w_1$, $v_1$, $\nabla w_1$, $w_2$.
$\nabla L=3w^2+2$.
$g_0=3(4)+2=14$, $v_0=196$, $w_1=2-\frac{0.5}{14}\cdot14=\mathbf{1.5}$.
$g_1=3(2.25)+2=\mathbf{8.75}$, $v_1=196+76.5625=\mathbf{272.5625}$.
$w_2=1.5-\frac{0.5}{\sqrt{272.56}}\cdot8.75=1.5-\frac{4.375}{16.51}=1.5-0.265=\mathbf{1.235}$ (range 1.08โ€“1.38).
ET Jan'26 ANComprehension Q3โ€“4 tricky
$f(x,y)=3(x^2+y^2)$, start $(x_0,y_0)\ne(0,0)$.
(i) GD, $\eta=0.1$, do steps: $(x_2,y_2)=\alpha(x_0,y_0)$. $\alpha$?
(ii) Ab AdaGrad ($v_{-1}=(0,0)$, $\epsilon=0$, $\eta=0.1$). Kya aisa constant $\alpha$ exist karta hai? Haan to $\alpha$, warna 0.
(i) $\nabla f=(6x,6y)$ โ†’ $x_{t+1}=x_t-0.6x_t=0.4x_t$ โ†’ $\alpha=0.4^2=\mathbf{0.16}$.
(ii) 0. AdaGrad pehle step pe: $v_0=36x_0^2$ โ†’ $x_1=x_0-0.1\cdot\frac{6x_0}{6|x_0|}=x_0-0.1\,\text{sign}(x_0)$. Har coordinate fixed amount (0.1) se khiskta hai, fixed fraction se nahi. To $x_2/x_0\ne y_2/y_0$ general me โ†’ koi common $\alpha$ nahi.
Seekh: AdaGrad (aur Adam) ka pehla step = $\eta\cdot\text{sign}(g)$ har coordinate me. GD direction preserve karta hai, AdaGrad nahi.

B. Moving average numerical

Quiz-2 Jan'25SA ยท 3 marks
$v_t=\beta v_{t-1}+(1-\beta)\nabla w_t$, $\beta=0.9$, $v_{-1}=0$, $\nabla w_0=5,\nabla w_1=3,\nabla w_2=4$. $v_2$?
$v_0=0.5$; $v_1=0.45+0.3=0.75$; $v_2=0.675+0.4=\mathbf{1.075}$ (range 1.065โ€“1.085).
Check: $v_2=0.1(0.81\cdot5+0.9\cdot3+4)=0.1(4.05+2.7+4)=1.075$ โœ”

C. Theory MCQ/MSQ

Quiz-2 Sep'24MSQ ยท 3 marks
Which optimization algorithms adapt the learning rate based on the current gradient and the history of gradients?
  1. GD with exponentially decaying learning rate
  2. RMSProp
  3. NAG with step decay
  4. ADAM
  5. Ada-Delta
Answer: (b) RMSProp, (d) ADAM, (e) Ada-Delta. Exponential decay / step decay sirf time (epoch) pe depend karte hain, gradient pe nahi.
Quiz-2 Sep'24MCQ ยท 2 marks
Which optimization algorithm combines the advantages of RMSprop and momentum?
  1. NAG
  2. Adam
  3. Adagrad
  4. Adadelta
Answer: (b) Adam.
Quiz-2 Sep'24MCQ ยท 2 marks
What is the key reason for the development of adaptive optimization algorithms like RMSprop and Adam?
  1. To increase the model size
  2. To adapt the learning rate for each parameter dynamically
  3. To reduce the number of layers
  4. To decay the initial learning rate
Answer: (b).
Quiz-2 Jan'25 ยท Sep'25 ยท PAMCQ ยท 3 marks asked 3ร—
Why is bias correction needed in the Adam optimizer?
  1. To prevent the learning rate from becoming too large in the early iterations
  2. To correct the underestimated moving averages of the first and second moments at the beginning of training
  3. To increase the momentum effect for faster convergence
  4. To ensure the learning rate remains constant throughout training
Answer: (b). $m_0=v_0=0$ se start โ†’ shuru me averages zero ki taraf biased. Fix: $\hat m_t=m_t/(1-\beta_1^t)$, $\hat v_t=v_t/(1-\beta_2^t)$.
โš  Option (a) tempting hai (slide me likha hai ki bina BC ke initial steps bade hote hain), lekin "need" ka official reason (b) hai.
Quiz-2 2025 T2MSQ ยท 3 marks
Which of the following optimizers use an adaptive learning rate?
  1. ADAM
  2. RMSprop
  3. Momentum based GD
  4. NAG
Answer: (a) ADAM, (b) RMSprop.
Quiz-2 Sep'25MSQ ยท 3 marks
Which of the following optimizers adapts the learning rate for each parameter during training?
  1. SGD
  2. NAG
  3. Adagrad
  4. RMSProp
Answer: (c) Adagrad, (d) RMSProp.
ET Sep'25 FNMSQ ยท 2 marks
Which optimizers use momentum (or momentum-like mechanisms)?
  1. NAG
  2. AdaGrad
  3. RMSProp
  4. AdaDelta
  5. ADAM
Answer: (a) NAG, (e) ADAM. RMSProp/AdaDelta me moving average squared gradient ka hai (denominator) โ€” woh momentum nahi hai. Momentum = gradient ($g$, square nahi) ka average.
Quiz-2 Jan'26MCQ ยท 1 mark
Which algorithm combines Nesterov accelerated gradient with adaptive moment estimation?
  1. NAdam
  2. Adam
  3. RMSprop with momentum
  4. NAG
Answer: (a) NAdam.
Quiz-2 Jan'26Matching ยท 4 marks important
A: 1. AdaGrad   2. RMSprop   3. Adam   4. AdaDelta
B: (i) Momentum + RMSprop, moving averages of gradient; (ii) LR scaled by total sum of all past squared gradients; (iii) exponentially decaying average of squared gradients to stop LR vanishing; (iv) bias-correction for initialization at origin; (v) adjusts updates by units of the parameters, no manual LR.
  1. 1-(i), 2-(v), 3-(ii, iii), 4-(iv)
  2. 1-(ii), 2-(iv), 3-(iii), 4-(i, v)
  3. 1-(ii), 2-(iii), 3-(i, iv), 4-(v)
  4. 1-(v), 2-(i), 3-(iv), 4-(ii, iii)
Answer: (c) 1-(ii), 2-(iii), 3-(i, iv), 4-(v). AdaDelta ka "units" wala point: numerator $\sqrt{u}$ me $\Delta w$ ki units hain, isliye update ki units parameter jaisi ho jaati hain aur $\eta$ ki zaroorat nahi.

D. LR schedule / line search

ET Sep'25 ANMCQ ยท 3 marks
$\eta=0.1$, batch 64 โ†’ 100 mini-batch updates/epoch. Step decay: LR halved every 150 updates. LR at end of 2nd epoch?
  1. 0.1
  2. 0.05
  3. 0.025
  4. 0.0125
Answer: (b) 0.05. 2 epochs = 200 updates โ†’ sirf 1 halving (150 pe). $\lfloor200/150\rfloor=1$.
Quiz-2 Jan'26SA ยท 3 marks
Backtracking line search: condition $J(w_{t+1})\le J(w_t)-10\eta$. Outcomes: $\eta=1.0$: $J+5$; $0.8$: $J-4$; $0.4$: $J-6$; $0.1$: $J-0.5$. Largest valid $\eta$?
Answer: 0.4. Required decrease $=10\eta$: 1.0 โ†’ cost badha โœ˜; 0.8 โ†’ chahiye 8, mila 4 โœ˜; 0.4 โ†’ chahiye 4, mila 6 โœ”; 0.1 โ†’ chahiye 1, mila 0.5 โœ˜.

15. Practice Assignment Week 5 (PA)

PA Q1 (a, b): AdaGrad
$v_0=0,\eta=1,\epsilon=0$; gradients 3, 4, 12. (a) Effective LR at iteration 3? (b) Iteration 4 se gradient 0 ho jaaye to?
(a) $v_3=9+16+144=169$ โ†’ $1/13=\mathbf{0.077}$ (key: 0.07).
(b) $v_t$ = 169 constant rehta hai, isliye $w$, $w_3$ pe hi rukta hai. ($v$ na badhta hai, na decay hota hai.)
PA Q2: RMSProp ka change kya karta hai?
Effective LR zero tak decay nahi hota aur low-gradient region me wapas badh sakta hai โ€” AdaGrad ka premature stalling avoid hota hai.
PA Q3: RMSProp, $\beta\in\{0.2,0.6,0.95\}$, kiska LR dheere decay?
Constant $g$ ke liye $v_t=(1-\beta^{t+1})g^2$. Bada ฮฒ โ†’ chhota $v_t$ โ†’ bada LR. Correct: ฮฒ=0.95 slower than ฮฒ=0.2 aur ฮฒ=0.95 slower than ฮฒ=0.6.
PA Q4: Adam me bias correction kyu?
Shuru me first aur second moments ke underestimated moving averages correct karne ke liye.
PA Q5: AdaMax & NAdam โ€” TRUE statements (MSQ)
  1. AdaMax L2 second moment ko $L_\infty$ se replace karta hai: $v_t=\max(\beta_2v_{t-1},|\nabla w_t|)$
  2. AdaMax me bias correction $m_t$ pe hai, $v_t$ pe nahi
  3. NAdam Adam ke first moment me Nesterov look-ahead daalta hai
  4. Zero gradients pe AdaMax ka denominator LR ko exponential averaging se zyada tezi se badhata hai
Answer: (a), (b), (c). (d) ulta hai โ€” max norm LR ko nahi badhata, exp. averaging badhata hai.
PA Q6: Cosine annealing $\eta_5$ ($\eta_{min}=0,\eta_{max}=1,T=10$)
$\cos(5\pi/10)=0$ โ†’ $\eta_5=\mathbf{0.5}$.
PA Q7: Cosine annealing vs exponential decay (saddle points)
Cosine annealing periodically LR ko $\eta_{max}$ pe restart karta hai โ†’ saddle points se escape. (Na yeh guarantee karta hai lower loss, na kam hyperparameters.)

16. Graded Assignment Week 5 (GA)

โš  GA ka official key PDF me nahi tha โ€” yeh answers maine khud solve/compute kiye hain (numericals Python se verify kiye). Submit karne se pehle ek baar khud bhi dekh lena.

GA Q1: Contours (fig a: ellipse $w_2$ ke along lamba; fig b: $w_1$ ke along lamba) โ€” sparse/dense
  1. fig a: $x_1$ sparse, $x_2$ dense
  2. fig a: $x_1$ dense, $x_2$ sparse
  3. fig b: $x_1$ dense, $x_2$ sparse
  4. fig b: $x_1$ sparse, $x_2$ dense
Answer: (b) and (d). Sparse feature โ†’ us weight me loss dheere badalta hai โ†’ contour us axis ke along stretched. Fig a vertically stretched โ†’ $w_2$ flat โ†’ $x_2$ sparse. Fig b horizontally stretched โ†’ $x_1$ sparse.
GA Q2: GD from $(-6,0)$ โ€” claim: trajectory seedhi horizontal line hogi minimum tak (fig a)
  1. True
  2. False
  3. Can't decide without running
Answer: (b) False. GD hamesha contour ke perpendicular chalta hai. Ellipse tilted hai, to $(-6,0)$ pe gradient ka $w_2$ component bhi non-zero โ†’ path seedha horizontal nahi hoga, curve karega.
GA Q3: Gradient profile (0.78 โ†’ 1 โ†’ 0.78 hump). AdaGrad ka effective LR kaunsa graph?
  1. U-shape (1.28 โ†’ 1 โ†’ 1.28)
  2. Tezi se girta hua curve (1.2 โ†’ 0.1)
  3. Badhta hua curve (1 โ†’ 9)
  4. Hump, gradient jaisa (0.078 โ†’ 0.1 โ†’ 0.078)
Answer: (b). $v_t$ hamesha badhta hai ((c) jaisa graph $\sqrt{v_t}$ ka hai) โ†’ $\eta/\sqrt{v_t}$ monotonically girta hai. Pehla point $\approx1/0.78=1.28$ โœ”.
GA Q4โ€“Q6: gradients $[1,0.9,0.6,0.01,0.1,0.2,0.5,0.55,0.56]$, $v_{-1}=0,\epsilon=0,\beta=0.9,\eta=0.1$
Q4: AdaGrad $\eta_6$?   Q5: $\eta_{100}>\eta_6$?   Q6: RMSProp $\eta_6$?
Q4: $v_6=1+0.81+0.36+0.0001+0.01+0.04+0.25=2.4701$ โ†’ $\eta_6=0.1/1.5717=\mathbf{0.06}$ (0.0636).
Q5: False โ€” AdaGrad me $v_t$ kabhi kam nahi hota, to $\eta_{100}\le\eta_6$.
Q6: RMSProp: $v_0=0.1,\ v_1=0.171,\ v_2=0.1899,\ v_3=0.1709,\ v_4=0.1548,\ v_5=0.1434,\ v_6=0.1540$ โ†’ $\eta_6=0.1/\sqrt{0.154}=\mathbf{0.25}$ (0.2548).
GA Q7: $L(w)\approx w^2$, $w_0=-2$, $\eta=0.1$, 10 iterations, Adam vs Momentum GD (ฮฒ=0.9)
  1. MGD moves past the minimum because of momentum
  2. Adam moves past the minimum because of momentum
  3. Adam doesn't move past the minimum because of adaptive LR
  4. MGD closer to minimum after 10 iterations
  5. Adam closer after 10 iterations
Simulation: MGD: โˆ’1.6, โˆ’0.92, โˆ’0.12, +0.62, 1.16, 1.42, 1.37, 1.05, 0.55, โˆ’0.009 โ†’ minimum cross karta hai aur 10 pe almost 0 pe.
Adam: โˆ’1.9, โˆ’1.8, โˆ’1.7, โ€ฆ, โˆ’1.02 (har step ~0.1, kyunki $\hat m/\sqrt{\hat v}\approx\pm1$).
Answer: (a), (c), (d).

17. My notes (khali pages)

Free writing space. Use โž• Page in the toolbar to add more ruled pages.