The Calculus Approach

The residual (error) made is

We try to find the vector of parameters such that we minimize

As in Calculus, for finding the minimum we take the derivative w.r.t. and set it to zero,

which is equivalent with the following system of normal equations