In the realm of machine learning, the cost function (or loss function) serves as a metric to gauge the alignment between a model’s predictions and the actual data.
Formula :
θnew = θold − α ⋅ ∇J(θ)
Where:
- θnew and θold represent the new and old parameter values (weights).
- α signifies the learning rate, influencing the magnitude of parameter adjustments.
- ∇J(θ) denotes the gradient of the cost function concerning the parameters.
- Compute the gradient (direction and rate of change) for each parameter within the cost function.
- Adjust the parameters in the reverse direction of the gradient to minimize the cost.
- Repeat until reaching the point of minimized cost function or its proximity.
(a) Batch Gradient Descent:
- Operation: Involves the entire training dataset to update parameters by computing the gradient for each iteration.
- Advantages: Ensures stability due to utilization of the complete dataset.
- Disadvantages: Tends to be sluggish with extensive datasets because it recomputes the gradient across the dataset at each step.
(b) Stochastic Gradient Descent (SGD):
- Operation: Deviates from leveraging the entire dataset, updating model parameters per training instance individually.
- Advantages: Swiftly handles substantial datasets.
- Disadvantages: Refined updates may trigger oscillations, impacting smooth convergence to the minimum.
(c) Mini-Batch Gradient Descent:
- Operation: Merges features of Batch and Stochastic Gradient Descent, updating model parameters with minor training instance packs.
- Advantages: More reliable than SGD and quicker than Batch Gradient Descent, extensively prevalent in practical applications, especially within deep learning domains.
- Disadvantages: Requires meticulous batch size tuning for optimal efficiency.
Gradient Descent can be enriched by diverse optimization strategies enhancing convergence speed and performance. These encompass:
(a) Momentum:
Momentum accelerates gradient descent by appending a fraction of the prior update to the current one, aiding the model in maneuvering ravines and averting oscillations by utilizing momentum from earlier steps.
v=βv+(1−β)∇J(θ)v
θ=θ−αv
(b) AdaGrad:
AdaGrad adapts the learning rate independently per parameter by scaling it reciprocally to the square root of the cumulative past gradients’ sum. This adaptation facilitates sparse data management but encounters diminishing learning rates in prolonged durations.
(c) RMSProp:
RMSProp advances upon AdaGrad, embedding adaptive learning rates for individual parameters while rectifying the vanishing learning rate situation through incorporating past gradients’ exponentially decaying average.
(d) Adam (Adaptive Moment Estimation):
Adam amalgamates both Momentum and RMSProp, calculating adaptive learning rates for each parameter contingent on both gradients’ mean (first moment) and variance (second moment). It enjoys vast application within deep learning landscapes.