What is Gradient Descent?
Gradient descent is an iterative procedure that minimizes the cost function parametrized by model parameters. It is an optimization method based on convex function and trims the parameters iteratively to help the given function attain its local minimum. Gradient measures the change in parameter with respect to the change in error. Imagine a blindfolded person on top of a hill and wanting to reach the lower altitude. The simple technique he can use is to feel the ground in every direction and take a step in the direction where the ground is descending faster. Here we need the help of the learning rate which says the size of the step we take to reach the minimum. The learning rate should be chosen so that it should not be too high or too low. When the selected learning rate is too high, it tends to bounce back and forth between the convex function of the gradient descent, and when it is too low, we will reach the minimum very slowly.
