# Gradient Descent

Last updated

Last updated

**Download pdf and code here**: https://github.com/Ex10si0n/Python-ML/blob/main/pyml_lab.zip?raw=true

Introduction

**Gradient descent** (GD) is an iterative first-order optimisation algorithm used to find a local minimum/maximum of a given function. This method is commonly used in *machine learning* (ML) and *deep learning*(DL) to minimise a cost/loss function (e.g. in a linear regression). Due to its importance and ease of implementation, this algorithm is usually taught at the beginning of almost all machine learning courses.

However, its use is not limited to ML/DL only, it’s being widely used also in areas like:

control engineering (robotics, chemical, etc.)

computer games

mechanical engineering

Implementing Gradient Descent Algorithms

In this Lab, we will implement a Linear Regression with Gradient Descent. Your task is to get the coefficients(k and b) of regression line via Gradient Descent. It might be a little bit hard, but don't worry, you just writing the codes following by given instructions.

Regression data and regression line

$\text{Regression Line}: y = 2.310601 \times x + 0.599747$

Procedure

**Program loss function:**

**Calculating Partial derivatives:**

**Applying Gradient Descent Algorithms:**

Start Programming

Basic Reqs. :

You need to replace the placeholders

`_____________________`

with your code, feel free to add new lines or helper methods.Once you have already finished, run the program and you will get the same output as mine.

Tips are given as comments in the code when necessary.

If you get stuck, feel free to ask me via Wechat or @me in Wechat Group or Email.

Submit your code

<your_name>_pyml_lab.pyor

<your_name>_pyml_lab.ipynbvia Wechat (ID:

ex10si0n-Yan) or Email ([email protected])

Extra self-learning (if you like):

Once you have finished, try the other

`x`

and`y`

you like.You can test the Linear Regression with any data you like such as: https://www.kaggle.com/quantbruce/real-estate-price-prediction, play it around and debug the model. I promise you will have an amazing discovery.

Also tries adjusting variable

`alpha`

(learning rate) like a Data Scientist. See what you get.Explore much more about Machine Learning at CS-229

Feel free to add more features in my code an turn it to your Project.

**Your output should be:**

This lab assignment is designed for let you get familiar with Gradient Descent. Once you correctly implemented the three `_____________`

's, I promise you will get same Regression Line as mine. If you do not get the Regression Line as mine, it is also fine to submit. You are welcomed to ask me for more tips.

Submission

Submit your code `<your_name>_pyml_lab.py`

**or** `<your_name>_pyml_lab.ipynb`

via Wechat (ID: `ex10si0n-Yan`

) or Email ([email protected])

$J(k, b) = \frac{1}{2}\times\sum\limits_{i=0}^{i < n}{(k \times x_i + b - y_i)^2}$

$\text{dk} \leftarrow \frac{\partial J(k, b)}{\partial k} = \sum_{i=0}^{i<n} [(kx_i + b - y_i) \times x_i]$

$\text{db} \leftarrow \frac{\partial J(k, b)}{\partial b} = \sum_{i=0}^{i<n} (kx_i + b - y_i)$

$\alpha = 0.0001$

$k \leftarrow k - \alpha \times \frac{\partial J(k, b)}{\partial{k}}$

$b \leftarrow b - \alpha \times \frac{\partial J(k, b)}{\partial{b}}$