Linear Least Square Regression uses a simple update: \[\theta _{next} = \theta _{current} - currentstepsize \cdot \theta _{current}^{'}\] \[currentstepsize = stepsize\cdot \frac{1}{\sqrt{iterIndex})}\]