```
| **List** | **Tuple** | **Set** | **Dictionary** |
| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| List is a non-homogeneous data structure which stores the elements in single row and multiple rows and columns | Tuple is also a non-homogeneous data structure which stores single row and multiple rows and columns | Set data structure is also non-homogeneous data structure but stores in single row | Dictionary is also a non-homogeneous data structure which stores key value pairs |
| List can be represented by \[ ] | Tuple can be represented by
( )
| Set can be represented by { } | Dictionary can be represented by { } |
| List allows duplicate elements | Tuple allows duplicate elements | Set will not allow duplicate elements | Set will not allow duplicate elements and dictionary doesn’t allow duplicate keys. |
| List can use nested among all | Tuple can use nested among all | Set can use nested among all | Dictionary can use nested among all |
| Example: \[1, 2, 3, 4, 5] | Example: (1, 2, 3, 4, 5) | Example: {1, 2, 3, 4, 5} | Example: {1, 2, 3, 4, 5} |
| List can be created using **list()** function | Tuple can be created using **tuple()** function. | Set can be created using **set()** function | Dictionary can be created using **dict()** function. |
| List is mutable i.e we can make any changes in list. | Tuple is immutable i.e we can not make any changes in tuple | Set is mutable i.e we can make any changes in set. But elements are not duplicated. | Dictionary is mutable. But Keys are not duplicated. |
| List is ordered | Tuple is ordered | Set is unordered | Dictionary is ordered |
| Creating an empty list
l=\[]
| Creating an empty Tuple
t=()
| Creating a set
a=set()
b=set(a)
| Creating an empty dictionary
d={}
|
To access a list item by index, the syntax is the same as Java.
```python
arr = [1, 2, 3, 4, 5]
print(arr[3]) # This will output 4
```
To work with dictionaries, we can use the key to get the value.
```python
dict = {"Alice": 12, "Bob": 13, "Charlie": 14}
print(dict["Alice"]) # This will output 12
```
### Loop and Condition
We want to print out all of the items in a list, we can simply use a for-each loop:
```python
arr = [1, 2, 3, 4, 5]
for a in arr:
print(a)
```
Or something similar as Java:
```python
arr = [1, 2, 3, 4, 5]
for i in range(len(arr)):
print(arr[i])
```
```java
int[] arr = {1, 2, 3, 4, 5};
for (int i = 0; i < arr.length; i++) {
System.out.println(arr[i]);
}
```
Or If we want to loop a dictionary:
```python
dict = {"Alice": 12, "Bob": 13, "Charlie": 14}
for key in dict:
print(key, dict[key])
```
`if` and `else` are very similar to that in Java:
```python
num = 10
if num >= 10:
print("num >= 10")
else:
print("num < 10")
```
```java
int num = 10;
if (num >= 10) {
System.out.println("num >= 10");
} else {
System.out.println("num < 10");
}
```
And Python can make checking easier:
```python
primes = [2, 3, 5, 7, 11, 13, 17, 19]
if 10 in primes:
print("10 is prime")
else:
print("10 is composite")
```
### Functions
Defining a function in Python is also similar to that in Java.
```python
def say_hello():
print("Hello")
```
```java
public static void sayHello() {
System.out.println("Hello");
}
```
Adding return value. And `if __name__ == '__main__':` means the main method.
```python
import math
def quadratic(a, b, c):
delta = b * b - 4 * a * c
if delta >=0:
root_A = ( -b - math.sqrt(delta) ) / 2 * a
root_B = ( -b + math.sqrt(delta) ) / 2 * a
return root_A, root_B
else:
return None
if __name__ == '__main__':
roots = quadratic(2, 3, 1)
print(roots)
```
### Try an Example
> 👍 Goal: Rewrite this Java code into Python code(script)
```java
class HelloWorld {
static boolean isPrime(int num) {
if (num == 0) return false;
if (num == 1) return false;
boolean flag = true;
for (int i = 2; i <= num / 2; i++) {
if (num % i == 0) {
flag = false;
break;
}
}
return flag;
}
public static void main(String[] args) {
for (int i = 0; i < 100; i++) {
if (isPrime(i)) System.out.println(i);
}
}
}
```
> Note that:
>
> * **Boolean** value in Python is `True` or `False`
### Activity: Clash of Code
{% embed url="" %}
# Object Oriented
Object-Oriented is also available in Python. Here is some simple code for illustrating OOP in Python.
```python
class Dog:
species = "Canis familiaris"
def __init__(self, name, age, breed):
self.name = name
self.age = age
self.breed = breed
def __str__(self):
return f"{self.name} is {self.age} years old"
def speak(self, sound):
print(f"{self.name} says :{sound}")
if __name__ == '__main__':
dogs = []
dogs.append(Dog("Miles", 4, "Jack Russell Terrier"))
dogs.append(Dog("Buddy", 9, "Dachshund"))
dogs.append(Dog("Jack", 3, "Bulldog"))
dogs.append(Dog("Jim", 5, "Bulldog"))
dogs[2].speak("I am hungry!")
print(dogs[1])
```
More about Object Oriented, please refer to COMP212 and COMP221.
# Importing modules
One feature of Python that makes it useful for a wide range of tasks is the fact that it comes "batteries included" – that is, the Python standard library contains useful tools for a wide range of tasks. On top of this, there is a broad ecosystem of third-party tools and packages that offer more specialized functionality. Here we'll take a look at importing standard library modules, tools for installing third-party modules, and a description of how you can make your own modules.
For loading built-in and third-party modules, Python provides the `import` statement. There are a few ways to use the statement, which we will briefly mention in this chapter, from most recommended to least recommended.
### Explicit module import
Explicit import of a module preserves the module's content in a namespace. The namespace is then used to refer to its contents with a "`.`" between them. For example, here we'll import the built-in `math` module and compute the cosine of pi:
```python
import math
math.cos(math.pi)
```
### Explicit module import by the alias
For longer module names, it's not convenient to use the full module name each time you access some element. For this reason, we'll commonly use the "`import ... as ...`" pattern to create a shorter alias for the namespace. For example, the NumPy (Numerical Python) package, a popular third-party package useful for data science, is by convention imported under the alias np:
```python
import numpy as np
np.cos(np.pi)
```
### Explicit import of module contents
Sometimes rather than importing the module namespace, you would just like to import a few particular items from the module. This can be done with the "`from ... import ...`" pattern. For example, we can import just the cos function and the pi constant from the math module:
```python
from math import cos, pi
cos(pi)
```
### Implicit import of module contents
Finally, it is sometimes useful to import the entirety of the module contents into the local namespace. This can be done with the "`from ... import *`" pattern:
```python
from math import *
sin(pi) ** 2 + cos(pi) ** 2
```
This pattern should be used sparingly, if at all. The problem is that such imports can sometimes overwrite function names that you do not intend to overwrite, and the implicitness of the statement makes it difficult to determine what has changed.
### Importing from the Python's standard library
Python's standard library contains many useful built-in modules, which you can read about fully in *Python's documentation*. Any of these can be imported with the `import` statement, and then explored using the help function seen on the previous page. Here is an extremely incomplete list of some of the modules you might wish to explore and learn about:
* `os` and `sys`: Tools for interfacing with the operating system, including navigating file directory structures and executing shell commands
* `math` and `cmath`: Mathematical functions and operations on real and complex numbers
* `itertools`: Tools for constructing and interacting with iterators and generators
* `functools`: Tools that assist with functional programming
* `random`: Tools for generating pseudorandom numbers
* `pickle`: Tools for object
* `persistence`: saving objects to and loading objects from the disk
* `json` and `csv`: Tools for reading JSON-formatted and CSV-formatted files.
* `urllib`: Tools for doing HTTP and other web requests.
### Importing from third-party modules
Installation in CLI.
```bash
pip install numpy
```
Import in code.
```python
import numpy as np
```
# pip: Package manager
`pip` is the [package installer](https://packaging.python.org/guides/tool-recommendations/) for Python. You can use pip to install packages from the [Python Package Index](https://pypi.org/) and other indexes.
The packages `pip` installed could be a Python module or executable binaries.
### Playaround
Let's have a look at `wikipedia` module, installation:
```bash
pip install wikipedia
```
Then we create a Python script with:
```python
import wikipedia
result = wikipedia.page("Macao Polytechnic Institute")
print(result.summary)
```
We can get:
```bash
Macao Polytechnic Institute (IPM; Chinese: 澳門理工學院; Portuguese: Instituto Politécnico de Macau) was established in 1981. It is located in the Macao Special Administrative Region of the People's Republic of China. MPI is a public Higher education institution with an emphasis on applied knowledge and skills.
```
# DS Utilities: Numpy
## TD; DR
In this Chapter, take the following 25 exercises to learn how to use the basic APIs of `numpy`.
***
## Exercise on Numpy
#### 1. Import the numpy package under the name `np`
```python
import numpy as np
```
#### 2. Create a null vector of size 10
```python
Z = np.zeros(10)
print(Z)
```
#### 3. Create a null vector of size 10 but the fifth value which is 1
```python
Z = np.zeros(10)
Z[4] = 1
print(Z)
```
#### 4. Create a vector with values ranging from 10 to 49
```python
Z = np.arange(10,50)
print(Z)
```
#### 5. Reverse a vector (first element becomes last)
```python
Z = np.arange(10)
Z = Z[::-1]
print(Z)
```
#### Also tries:
```python
Z = np.arange(10)
Z = Z[::-2]
print(Z)
```
```python
Z = np.arange(10)
Z = Z[::3]
print(Z)
```
#### 6. Create a 3x3 matrix with values ranging from 0 to 8
```python
Z = np.arange(9).reshape(3, 3)
print(Z)
```
#### 7. Find indices of non-zero elements from `[1,2,0,0,4,0]`
```python
nz = np.nonzero([1,2,0,0,4,0])
print(nz)
```
#### 8. Create a 3x3 identity matrix
> Identity matrix: values on **positive diagnosis** of matrix are 1, and others are 0
>
> Also recognized as I\_n
```python
Z = np.eye(3)
print(Z)
```
#### 9. Create a 3x3x3 array with random values
```python
Z = np.random.random((3,3,3))
print(Z)
```
#### 10. Create a 10x10 array with random values and find the minimum and maximum values
```python
Z = np.random.random((10,10))
Zmin, Zmax = Z.min(), Z.max()
print(Zmin, Zmax)
```
#### 11. Create a random vector of size 30 and find the mean value
```python
Z = np.random.random(30)
m = Z.mean()
print(m)
```
#### 12. Create a 2d array with 1 on the border and 0 inside
```python
Z = np.ones((10,10))
Z[1:-1,1:-1] = 0
print(Z)
```
#### 13. How to add a border (filled with 0's) around an existing array?
```python
Z = np.arange(25).reshape(5, 5)
Z = np.pad(Z, pad_width=1, mode='constant', constant_values=0)
print(Z)
```
#### 14. Create a 5x5 matrix with values 0,1,2,3,4 on the diagonal
```python
Z = np.diag(np.arange(5))
print(Z)
```
#### 15. Create a 8x8 matrix and fill it with a chess board pattern
> Chess board Pattern is just like:
>
> ```
> 0 1 0 1 0 1 0 1
> 1 0 1 0 1 0 1 0
> 0 1 0 1 0 1 0 1
> 1 0 1 0 1 0 1 0
> 0 1 0 1 0 1 0 1
> 1 0 1 0 1 0 1 0
> 0 1 0 1 0 1 0 1
> 1 0 1 0 1 0 1 0
> ```
```python
Z = np.zeros((8,8),dtype=int)
Z[1::2,::2] = 1
Z[::2,1::2] = 1
print(Z)
# Alternative
Z = np.tile(np.array([[0,1],[1,0]]), (4,4))
print(Z)
```
#### 16. Consider a (6,7,8) shape array, what is the index (x,y,z) of the 100th element?
```python
print(np.unravel_index(99,(6,7,8)))
```
#### 17. Matrix Addition and Subtraction
Add up two matries A and B, where A and B respectively be:
```
A = [[1 2 3]
[4 5 6]
[7 8 9]]
B = [[9 8 7]
[6 5 4]
[3 2 1]]
```
```python
A = np.arange(1, 10).reshape(3, 3)
B = np.arange(9, 0, -1).reshape(3, 3)
print(A + B)
```
#### 18. Normalize a 5x5 random matrix
> **Normalize a matrix**(归一化矩阵) is to make every values in the matrix lies on \[min, max], as an example: we have an array valued \[-1, 1, 3]. After normalization, it becomes \[0, 0.5, 1].
```python
Z = np.random.random((5, 5))
Z = (Z - Z.min())/(Z.max() - Z.min())
print(Z)
```
#### 19. Multiply a 5x3 matrix by a 3x2 matrix (real matrix product)
```python
Z = np.dot(np.ones((5,3)), np.ones((3,2)))
print(Z)
```
#### 20. Given a 1D array, negate all elements which are between 3 and 8, in place.
```python
Z = np.arange(11)
Z[(3 < Z) & (Z < 8)] *= -1
print(Z)
```
#### 21. How to find common values between two arrays?
```python
Z1 = np.random.randint(0,10,10)
Z2 = np.random.randint(0,10,10)
print(Z1, Z2)
print(np.intersect1d(Z1,Z2))
```
#### 22. How to get the dates of yesterday, today and tomorrow?
```python
yesterday = np.datetime64('today') - np.timedelta64(1)
today = np.datetime64('today')
tomorrow = np.datetime64('today') + np.timedelta64(1)
print(yesterday, today, tomorrow)
```
#### 23. Create a vector of size 6 with values in equal spacing ranges \[0, 20]
```python
Z = np.linspace(0,20,6)
print(Z)
```
#### 24. Create a random vector of size 10 and sort it
```python
Z = np.random.random(10)
Z.sort()
print(Z)
```
#### 25. Create random vector of size 10 and replace the maximum value by 0
```python
Z = np.random.random(10)
Z[Z.argmax()] = 0
print(Z)
```
> What is the relationship between `Z.argmax()` and `Z.max()`?
>
> ```python
> Z = np.random.random(10)
> print(Z[Z.argmax()] == Z.max())
> ```
# Implementation of Neural Networks from scratch
TensorFlow Playground: [click here](https://playground.tensorflow.org/#activation=tanh\&batchSize=30\&dataset=xor\®Dataset=reg-plane\&learningRate=0.03\®ularizationRate=0\&noise=40\&networkShape=4,2\&seed=0.05981\&showTestData=false\&discretize=false\&percTrainData=60\&x=true\&y=true\&xTimesY=false\&xSquared=false\&ySquared=false\&cosX=false\&sinX=false\&cosY=false\&sinY=false\&collectStats=false\&problem=classification\&initZero=false\&hideText=false)
## Introduction
In the past 10 years, the best-performing artificial-intelligence systems — such as the speech recognizers on smartphones or Google’s latest automatic translator — have resulted from a technique called “Deep Learning.”
Deep learning is in fact a new name for an approach to artificial intelligence called neural networks, which have been going in and out of fashion for more than 70 years. **Neural networks** were first proposed in 1944 by Warren McCullough and Walter Pitts.
**Neural network** is an interconnected group of nodes, inspired by a simplification of neurons in a brain. It works similarly to the human brain’s neural network. A “neuron” in a neural network is a mathematical function that collects and classifies information according to a specific architecture. The network bears a strong resemblance to statistical methods such as curve fitting and regression analysis.
**In this course**, we’ll understand how neural networks work while implementing one from scratch in Python.
In the previous chapter, we have already learned one of the Data Science Utilities `numpy` from [25 exercises to get familiar with Numpy](https://pyml.aspires.cc/chapter2/data-science-utilities). Since you have learned it yet, we have enough knowledge to learn how to **implement** a Neural Network only using `numpy`. This lecture is inspired from [VictorZhou's Blog](https://github.com/Ex10si0n/Python-ML/blob/main/chapter3/victorzhou.com).
## Neuron
### Biology Stuffs
In field of study in **Neuroscience and Biology**, scientists have already built the model of a Neuron (Nerve cell) in creatures brains. a Neuron is illustrated as follow:
Now focus on these 3 components:
* **Dendrites(树突)**: Receive signals from other cells.
* **Axon hillock(轴丘)**: Generates impulse in the neuron.
* **Axon terminal(轴突末梢)**: Forms junctions with other cells.
At the **Dendrites(树突)**, **neurotransmitters(神经递质)** are released, triggering **nerve impulse(神经冲动)** in neighboring neurons. When the neighboring nerve cell(s) receives a specific impulse, the cell body will be **activated** and **transmit electronic signal** through the **Axon(轴突)** by the effect of exchanging electrons with **Na+/K+-ATPase(Na/K泵)**.
Remember, a neuron is always connected to many other neighboring neurons in the **Axon terminal(轴突末梢) side**, meanwhile; a neuron can also receive impluse from previous neurons in the **Dendrites(树突) side**. As an example:
### Metaphor: Interpersonal Relationship
> There is a boy called Tom, he has a simple **interpersonal relationship** described as the following picture:
>
> 
>
> **Henderson and Alice** are Tom's good friends while **Jackie** is Tom's colleague, it is clear that Tom will arrange higher priorities with affairs from Henderson and Alice, and a lower priorities to Jackie. Relatively, **Rose** regards Tom as good friend so when Tom tells Rose something, she will be more active than **Alex** in helping Tom.
>
> Years later, with the co-operation with Jackie, Tom and Jackie will be more close and be good friend, they talk to each other everything. So the connection between Jackie and Tom will the stronger, and Tom will arrange higher priorities to Jackie.
>
> Tom also have a **threshold** on which kind of people he can help and which kind of people he will reject. Meanwhile, the **threshold** is always changing for each people. Maybe someday, Tom and Alice have not communicated with each other for several month, and Tom will regards Alice not as close as before, so he will adjust the **threshold**.
Back to neurons, the relationships is similar. With multiple neurons work as the **interpersonal relationship**, they are forming a network like **social networks**. And that is why our brains can think, action and create as well as having emotion.
Now let us model the **working principle** of neurons with programming language. Image a function which can take two input parameters as **x1, x2** and returns output **y**. In Python code:
```python
def f(x1, x2):
...
return y
```
The implementation `...` of the function are in three steps:
1. Calculate **weighted** x1 and x2
2. Add a **bias** to adjust the **weighted value** from step 1
3. Determine in what extents we can **accept** the value from step 2
To illustrate the implementation, we have:
```python
def f(x1, x2):
w = w1 * x1 + w2 * x2
expr = w + b
y = activation(expr)
return y
```
Note that, **weights** w1, w2 and **bias** b can be changed during the time. And `accept_level` can be considered as the **threshold** to determine how much **y** will be.
This function works as a **neuron** which we have introduced before. There will be many neurons interconnected as a network.
In this example, we regards:
* **x1, x2** as the output from the previous neuron.
* **y** as the output of current function.
* `activation()` is the activation function which used to turn an unbounded input into an output that has a nice, predictable form.
### Activation Function
> #### Activation function
>
> Consider a classification problem: You are taking a test at school, if your test mark is lower than **50**, your mama will **beat you** when you back home. Otherwise, she will **treate you a delicious dinner(请你吃顿好的)**, we can consider
>
> * `x = your mark - 50`
> * when x is positive, you will have a delicious dinner
> * when x is negative, you will be beaten by mama
>
> That is the following diagram:
>
> 
>
> * `y = 1` means you have dinner
> * `y = 0` means you got beat
>
> But it is not fair for your score is around 50, which means only 1 mark difference will **make you be in two results**. Why not make the line smooth, and when you got whatever 49 or 51 your mama will just say: It's okay, be better next time.
>
> **Sigmoid activation function** works well in this scenario:
>
> 
>
> Back to the activation function, actually there are many kinds of functions and they work well in different senarios.
>
> 
### Coding Neuron
In order to manage the relationship well. We can make a `Neuron` class to struturalize a neuron unit. In which, the `feedforward()` function is the previous `f(x1, x2)`. And inner implementation of the `f()` function changes in order to get **vector input** `inputs`, so it will be `np.dot()` instead of `w1 * x1 + w2 * x2`. If you are not familiar with dot product, please refer: [Dot product in Wiki](https://simple.wikipedia.org/wiki/Dot_product).
```python
def sigmoid(x):
# Activation function: f(x) = 1 / (1 + e^(-x))
return 1 / (1 + np.exp(-x))
class Neuron:
def __init__(self, weights, bias):
self.weights = weights
self.bias = bias
def feedforward(self, inputs):
# Weight inputs, add bias, then use the activation function
total = np.dot(self.weights, inputs) + self.bias
return sigmoid(total)
```
## Neuron Network
A neural network is nothing more than a bunch of neurons connected together. Here’s what a simple neural network might look like:
This network has 2 inputs, a **hidden layer** with 2 neurons (**h1** and **h2**), and an **output layer** with 1 neuron (**o1**). Notice that the inputs for **o1** are the outputs from **h1** and **h2** - that’s what makes this a network. In addition, **h1, h2, o1** are all **instances** of class Neuron.
> *A* **hidden layer** is any layer between the input (first) layer and output (last) layer. There can be multiple hidden layers!
### Feed-forward
We assumed that all of the Neurons (h1, h2, o1) have the same weights `w = [0, 1]` and bias `b = 0`. Then if we pass input `x1 = 2, x2 = 3` into the network. We have:
A neural network can have **any number of layers** with **any number of neurons** in those layers. The basic idea stays the same: feed the input(s) forward through the neurons in the network to get the output(s) at the end.
### Coding Feed-forward
Here is the code implementing the previous feedforward process of our neural network.
```python
import numpy as np
def sigmoid():
# Omitted...
class Neuron:
# Omitted...
class OurNeuralNetwork:
'''
A neural network with:
- 2 inputs
- a hidden layer with 2 neurons (h1, h2)
- an output layer with 1 neuron (o1)
Each neuron has the same weights and bias:
- w = [0, 1]
- b = 0
'''
def __init__(self):
weights = np.array([0, 1])
bias = 0
# The Neuron class here is from the previous section
self.h1 = Neuron(weights, bias)
self.h2 = Neuron(weights, bias)
self.o1 = Neuron(weights, bias)
def feedforward(self, x):
out_h1 = self.h1.feedforward(x)
out_h2 = self.h2.feedforward(x)
# The inputs for o1 are the outputs from h1 and h2
out_o1 = self.o1.feedforward(np.array([out_h1, out_h2]))
return out_o1
network = OurNeuralNetwork()
x = np.array([2, 3])
print(network.feedforward(x)) # 0.7216325609518421
```
We got the same result compared with the illustrated diagram in Feed-Forward section.
### Training a Neural Network (Part 1)
Say we have the following measurements:
| Name | Weight (lb) | Height (in) | Gender |
| ------- | ----------- | ----------- | ------ |
| Alice | 133 | 65 | F |
| Bob | 160 | 72 | M |
| Charlie | 152 | 70 | M |
| Diana | 120 | 60 | F |
Let’s train our network to predict someone’s gender given their weight and height:
We’ll represent Male with a 0 and Female with a 1, and we’ll also shift the data to make it easier to use:
make it easier to use:
| Name | Weight (minus 135) | Height (minus 66) | Gender |
| ------- | ------------------ | ----------------- | ------ |
| Alice | -2 | -1 | 1 |
| Bob | 25 | 6 | 0 |
| Charlie | 17 | 4 | 0 |
| Diana | -15 | -6 | 1 |
> I arbitrarily chose the shift amounts (135 and 66) to make the numbers look nice. Normally, you’d shift by the mean.
>
> The process making data more easier to use follows the methodology of **Normalization**.
>
> 
#### Linear Regression
Before we start the training process, let's have a quick review on **Linear Regression**:
Consider a plot of scatter points `(x_i, y_i)`which repectively represents `(rental price, house area)`. As common sense, the **x** and **y** has a positive correlation. We can plug in each **x\_i** and **y\_i** into the formula of **Least Squares Method**:
Where `m` called **correlation coefficient** and the equation of the regression line is:
For linear regression, we can get the equation easily. But for **Multiple Linear Regression** with many parameter **x**'s, it will become complex.
And we know that, to predict the **house rental price**, we cannot consider only one factor which is house area. We should also take **Location**, **Number of Bedrooms**, **Appliances and Other Amenities**, **Curb Appeal and Condition** and even if the house is **Allowing Pets**. Then the regression process will be further more complex. So we should determine another method to optimize the calculating time.
#### Loss
Let us have a look at a common component of regression. We know that any of the avaliable regression cannot pass through all of the points of (x, y). There may be many points below line or on the upwards. So the **Loss** is defined to describe the regression's level of inaccurate.
The **red lines** describe absolute value of `y_accurate - y_predict`. As an example, we use the **mean squared error** (MSE) loss:
Turn this formula to Python code is straightforward:
```python
import numpy as np
def mse_loss(y_true, y_pred):
return ((y_true - y_pred) ** 2).mean()
y_true = np.array([1, 0, 0, 1])
y_pred = np.array([0, 0, 0, 0])
print(mse_loss(y_true, y_pred)) # 0.5
```
#### Gradient Descent
Assume that The **z** axis (height) is the loss, **x** and **y** are the correlation coeffiecients of a **Multiple Linear Regression**. There may be a 3D plot as follows.
To get the optimized regression effect. We should determine the **lowest z** on the plot and the coefficients are the corresponding **x** and **y**. If we can get this plot, we can easily find where is the **lowest z**.
Unfortunately, in practice, like playing a open-world game. It always has fog covering the unvisited area. We only have the information of starting up point, and we need to try each direction to explore.
But, at least, we may know where to go like: **try not going to the desert at first.**
Same as calculating the **optimized correlation coefficient**. Although we do not know all of the map, we can refer to the gradient where we are standing on. We can refer **gradient(梯度)** as **slope(斜率)** in higher dimension plot. As the following animation illustrated, if we want to go to the lower place, we should follow the current gradient:
> **Positive** gradient: go **Left**
>
> **Negative** gradient: go **Right**
### Training a Neural Network (Part 2)
We now have a clear goal: **minimize the loss** of the neural network. We know we can change the network’s **weights** and **biases** (each weights and bias in a Neuron) to influence its predictions, but how do we do so in a way that decreases loss?
To think about loss is as a function of weights and biases. Let’s label each weight and bias in our network:
Then the loss of the whole Network are determined by weight and bias in each Neurons:
#### Backpropagation
If we want to make **w1** a little bit higher or lower, how would **Loss** change? The loss is the function of every weights and biases. So we need to calculate the gradient of the 2D slice `(x, y) = (w1, L)` of a 9D plot is [partial derivative](https://simple.wikipedia.org/wiki/Partial_derivative) .
This system of calculating partial derivatives by working backwards is known as **backpropagation**, or “backprop”.
#### Optimizer
**We have all the tools we need to train a neural network now!** We’ll use an optimization algorithm called [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation:
η is a constant called the **learning rate** that controls how fast we train. We use η times to adjusting **w1**:
* if  is positive, according to SGD **w1** will decrease, `L = f(w1)` is **decrescent function**, **L** will decrease
* if  is negative, according to SGD, **w1** will increase, `L = f(w1)` is **increasing function**, **L** will decrease
If we do this for every weight and bias in the network, the loss will slowly decrease and our network will improve.
Our training process will look like this:
1. Choose **one** sample from our dataset. This is what makes it *stochastic*gradient descent - we only operate on one sample at a time.
2. Calculate all the partial derivatives of loss (example using , but we need to find **all w and b**) with respect to **weights** or **biases**
3. Use the update equation to update each **weight** and **bias**.
4. Go back to step 1.
Implementing with Python is, we refer each **w** and **b** is random:
```python
import numpy as np
def sigmoid(x):
# Sigmoid activation function: f(x) = 1 / (1 + e^(-x))
return 1 / (1 + np.exp(-x))
def deriv_sigmoid(x):
# Derivative of sigmoid: f'(x) = f(x) * (1 - f(x))
fx = sigmoid(x)
return fx * (1 - fx)
def mse_loss(y_true, y_pred):
# y_true and y_pred are numpy arrays of the same length.
return ((y_true - y_pred) ** 2).mean()
class OurNeuralNetwork:
'''
A neural network with:
- 2 inputs
- a hidden layer with 2 neurons (h1, h2)
- an output layer with 1 neuron (o1)
*** DISCLAIMER ***:
The code below is intended to be simple and educational, NOT optimal.
Real neural net code looks nothing like this. DO NOT use this code.
Instead, read/run it to understand how this specific network works.
'''
def __init__(self):
# Weights
self.w1 = np.random.normal()
self.w2 = np.random.normal()
self.w3 = np.random.normal()
self.w4 = np.random.normal()
self.w5 = np.random.normal()
self.w6 = np.random.normal()
# Biases
self.b1 = np.random.normal()
self.b2 = np.random.normal()
self.b3 = np.random.normal()
def feedforward(self, x):
# x is a numpy array with 2 elements.
h1 = sigmoid(self.w1 * x[0] + self.w2 * x[1] + self.b1)
h2 = sigmoid(self.w3 * x[0] + self.w4 * x[1] + self.b2)
o1 = sigmoid(self.w5 * h1 + self.w6 * h2 + self.b3)
return o1
def train(self, data, all_y_trues):
'''
- data is a (n x 2) numpy array, n = # of samples in the dataset.
- all_y_trues is a numpy array with n elements.
Elements in all_y_trues correspond to those in data.
'''
learn_rate = 0.1
epochs = 1000 # number of times to loop through the entire dataset
for epoch in range(epochs):
for x, y_true in zip(data, all_y_trues):
# --- Do a feedforward (we'll need these values later)
sum_h1 = self.w1 * x[0] + self.w2 * x[1] + self.b1
h1 = sigmoid(sum_h1)
sum_h2 = self.w3 * x[0] + self.w4 * x[1] + self.b2
h2 = sigmoid(sum_h2)
sum_o1 = self.w5 * h1 + self.w6 * h2 + self.b3
o1 = sigmoid(sum_o1)
y_pred = o1
# --- Calculate partial derivatives.
# --- Naming: d_L_d_w1 represents "partial L / partial w1"
d_L_d_ypred = -2 * (y_true - y_pred)
# Neuron o1
d_ypred_d_w5 = h1 * deriv_sigmoid(sum_o1)
d_ypred_d_w6 = h2 * deriv_sigmoid(sum_o1)
d_ypred_d_b3 = deriv_sigmoid(sum_o1)
d_ypred_d_h1 = self.w5 * deriv_sigmoid(sum_o1)
d_ypred_d_h2 = self.w6 * deriv_sigmoid(sum_o1)
# Neuron h1
d_h1_d_w1 = x[0] * deriv_sigmoid(sum_h1)
d_h1_d_w2 = x[1] * deriv_sigmoid(sum_h1)
d_h1_d_b1 = deriv_sigmoid(sum_h1)
# Neuron h2
d_h2_d_w3 = x[0] * deriv_sigmoid(sum_h2)
d_h2_d_w4 = x[1] * deriv_sigmoid(sum_h2)
d_h2_d_b2 = deriv_sigmoid(sum_h2)
# --- Update weights and biases
# Neuron h1
self.w1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w1
self.w2 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w2
self.b1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_b1
# Neuron h2
self.w3 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w3
self.w4 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w4
self.b2 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_b2
# Neuron o1
self.w5 -= learn_rate * d_L_d_ypred * d_ypred_d_w5
self.w6 -= learn_rate * d_L_d_ypred * d_ypred_d_w6
self.b3 -= learn_rate * d_L_d_ypred * d_ypred_d_b3
# --- Calculate total loss at the end of each epoch
if epoch % 10 == 0:
y_preds = np.apply_along_axis(self.feedforward, 1, data)
loss = mse_loss(all_y_trues, y_preds)
print("Epoch %d loss: %.3f" % (epoch, loss))
```
Then we input some train data from the following table.
| Name | Weight (minus 135) | Height (minus 66) | Gender |
| ------- | ------------------ | ----------------- | ------ |
| Alice | -2 | -1 | 1 |
| Bob | 25 | 6 | 0 |
| Charlie | 17 | 4 | 0 |
| Diana | -15 | -6 | 1 |
```python
# Define dataset
data = np.array([
[-2, -1], # Alice
[25, 6], # Bob
[17, 4], # Charlie
[-15, -6], # Diana
])
all_y_trues = np.array([
1, # Alice
0, # Bob
0, # Charlie
1, # Diana
])
# Train our neural network!
network = OurNeuralNetwork()
network.train(data, all_y_trues)
```
Our loss steadily decreases as the network learns:
We can now use the network to predict genders:
```python
# Make some predictions
emily = np.array([-7, -3]) # 128 pounds, 63 inches
frank = np.array([20, 2]) # 155 pounds, 68 inches
print("Emily: %.3f" % network.feedforward(emily)) # 0.951 - F
print("Frank: %.3f" % network.feedforward(frank)) # 0.039 - M
```
Try your personal data:
```python
def generate_test_data(kg, cm):
lbs = kg * 2.20462
inch = cm * 0.393701
return np.array([lbs - 135, inch - 66])
steve = generate_test_data(182, 73)
print("Steve: %.3f" % network.feedforward(steve)) # 0.040 - M
```
## Conclusion
You made it! A quick review of what we did:
* Introduced **neurons**, the building blocks of neural networks.
* Used the **sigmoid activation function** in our neurons.
* Saw that neural networks are just neurons connected together.
* Created a dataset with Weight and Height as inputs (or **features**) and Gender as the output (or **label**).
* Learned about **loss functions** and the **mean squared error** (MSE) loss.
* Realized that training a network is just minimizing its loss.
* Used **backpropagation** to calculate partial derivatives.
* Used **stochastic gradient descent** (SGD) to train our network.
Thanks for reading!
# Gradient Descent
**Download pdf and code here**: [https://github.com/Ex10si0n/Python-ML/blob/main/pyml\_lab.zip?raw=true](https://github.com/Ex10si0n/Python-ML/blob/main/pyml_lab.zip)
### Introduction
**Gradient descent** (GD) is an iterative first-order optimisation algorithm used to find a local minimum/maximum of a given function. This method is commonly used in *machine learning* (ML) and *deep learning*(DL) to minimise a cost/loss function (e.g. in a linear regression). Due to its importance and ease of implementation, this algorithm is usually taught at the beginning of almost all machine learning courses.
However, its use is not limited to ML/DL only, it’s being widely used also in areas like:
* control engineering (robotics, chemical, etc.)
* computer games
* mechanical engineering
### Implementing Gradient Descent Algorithms
In this Lab, we will implement a Linear Regression with Gradient Descent. Your task is to get the coefficients(k and b) of regression line via Gradient Descent. It might be a little bit hard, but don't worry, you just writing the codes following by given instructions.
#### Regression data and regression line
```
x = [1, 2, 4, 4, 5, 6, 9, 9, 9, 10]y = [4, 5, 8, 12, 15, 16, 18, 21, 22, 24]
```
$$
\text{Regression Line}: y = 2.310601 \times x + 0.599747
$$
### Procedure
**Program loss function:**
$$
J(k, b) = \frac{1}{2}\times\sum\limits\_{i=0}^{i < n}{(k \times x\_i + b - y\_i)^2}
$$
**Calculating Partial derivatives:**
$$
\text{dk} \leftarrow \frac{\partial J(k, b)}{\partial k} = \sum\_{i=0}^{i\ **Basic Reqs. :**
>
> * You need to replace the placeholders `_____________________` with your code, feel free to add new lines or helper methods.
> * Once you have already finished, run the program and you will get the same output as mine.
> * Tips are given as comments in the code when necessary.
> * If you get stuck, feel free to ask me via Wechat or @me in Wechat Group or Email.
> * **Submit your code `_pyml_lab.py` or `_pyml_lab.ipynb` via Wechat (ID: `ex10si0n-Yan`) or Email (**[**p1908326@ipm.edu.mo**](mailto:p1908326@ipm.edu.mo)**)**
>
> **Extra self-learning (if you like):**
>
> * Once you have finished, try the other `x` and `y` you like.
> * You can test the Linear Regression with any data you like such as: , play it around and debug the model. I promise you will have an amazing discovery.
> * Also tries adjusting variable `alpha` (learning rate) like a Data Scientist. See what you get.
> * Explore much more about Machine Learning at [CS-229](https://cs229.stanford.edu/syllabus.html)
> * Feel free to add more features in my code an turn it to your Project.
```python
class LinearRegression:
def __init__(self, x, y):
self.x = x
self.y = y
self.k = 0
self.b = 0
self.J = self.loss(self.k, self.b)
self.J_history = []
def loss(self, k, b):
# Tips here:
# program the loss function with given formula
# where x is self.x, y is self.y from class LinearRegression
# which means you can get x_i by for loop all of the indices of 0 ... len(x)
loss = _____________________
return loss
def gradient_descent(self, alpha, iterations):
for i in range(iterations):
# Tips here:
# program the partial derivatives dk, db
# refer the formula from section [Calculating Partial derivatives]
# where k is self.k, b is self.b from class LinearRegression
# notice that the code here is already inside for loop
dk = _____________________
db = _____________________
self.k = self.k - alpha * dk
self.b = self.b - alpha * db
self.J_history.append(self.loss(self.k, self.b))
return self.k, self.b
def predict(self, x):
return self.k * x + self.b
if __name__ == '__main__':
x = [1, 2, 4, 4, 5, 6, 9, 9, 9, 10]
y = [4, 5, 8, 12, 15, 16, 18, 21, 22, 24]
model = LinearRegression(x, y)
model.gradient_descent(0.0001, 1000)
print('Regression Line: y = %f * x + %f' % (model.k, model.b))
```
**Your output should be:**
This lab assignment is designed for let you get familiar with Gradient Descent. Once you correctly implemented the three `_____________`'s, I promise you will get same Regression Line as mine. If you do not get the Regression Line as mine, it is also fine to submit. You are welcomed to ask me for more tips.
```
Regression Line: y = 2.309133 * x + 0.610665
```
### Submission
Submit your code `_pyml_lab.py` **or** `_pyml_lab.ipynb`
via Wechat (ID: `ex10si0n-Yan`) or Email ()
# Introduction to Tensorflow
Copyright 2019 The TensorFlow Authors
```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```
## TensorFlow 2 quickstart for beginners
| [View on TensorFlow.org](https://www.tensorflow.org/tutorials/quickstart/beginner) | [Run in Google Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb) | [View source on GitHub](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb) | [Download notebook](https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/quickstart/beginner.ipynb) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
This short introduction uses [Keras](https://www.tensorflow.org/guide/keras/overview) to:
1. Load a prebuilt dataset.
2. Build a neural network machine learning model that classifies images.
3. Train this neural network.
4. Evaluate the accuracy of the model.
This tutorial is a [Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb) notebook. Python programs are run directly in the browser—a great way to learn and use TensorFlow. To follow this tutorial, run the notebook in Google Colab by clicking the button at the top of this page.
1. In Colab, connect to a Python runtime: At the top-right of the menu bar, select *CONNECT*.
2. Run all the notebook code cells: Select *Runtime* > *Run all*.
### Set up TensorFlow
```
import tensorflow as tf
print("TensorFlow version:", tf.__version__)
```
If you are following along in your own development environment, rather than [Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb), see the [install guide](https://www.tensorflow.org/install) for setting up TensorFlow for development.
Note: Make sure you have upgraded to the latest `pip` to install the TensorFlow 2 package if you are using your own development environment. See the [install guide](https://www.tensorflow.org/install)for details.
### Load a dataset
```
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
```
### Build a machine learning model
```
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
```
For each example, the model returns a vector of [logits](https://developers.google.com/machine-learning/glossary#logits) or [log-odds](https://developers.google.com/machine-learning/glossary#log-odds) scores, one for each class.
```
predictions = model(x_train[:1]).numpy()
predictions
```
The `tf.nn.softmax` function converts these logits to *probabilities* for each class:
```
tf.nn.softmax(predictions).numpy()
```
Note: It is possible to bake the `tf.nn.softmax` function into the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to provide an exact and numerically stable loss calculation for all models when using a softmax output.
Define a loss function for training using `losses.SparseCategoricalCrossentropy`, which takes a vector of logits and a `True` index and returns a scalar loss for each example.
```
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
```
This loss is equal to the negative log probability of the true class: The loss is zero if the model is sure of the correct class.
This untrained model gives probabilities close to random (1/10 for each class), so the initial loss should be close to `-tf.math.log(1/10) ~= 2.3`.
```
loss_fn(y_train[:1], predictions).numpy()
```
Before you start training, configure and compile the model using Keras `Model.compile`. Set the [`optimizer`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers) class to `adam`, set the `loss` to the `loss_fn`function you defined earlier, and specify a metric to be evaluated for the model by setting the `metrics` parameter to `accuracy`.
```
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])
```
### Train and evaluate your model
```
model.fit(x_train, y_train, epochs=5)
```
The `Model.evaluate` method checks the models performance, usually on a "[Validation-set](https://developers.google.com/machine-learning/glossary#validation-set)" or "[Test-set](https://developers.google.com/machine-learning/glossary#test-set)".
```
model.evaluate(x_test, y_test, verbose=2)
```
The image classifier is now trained to \~98% accuracy on this dataset. To learn more, read the [TensorFlow tutorials](https://www.tensorflow.org/tutorials/).
If you want your model to return a probability, you can wrap the trained model, and attach the softmax to it:
```
probability_model = tf.keras.Sequential([
model,
tf.keras.layers.Softmax()
])
```
```
probability_model(x_test[:5])
```
### Conclusion
Congratulations! You have trained a machine learning model using a prebuilt dataset using the [Keras](https://www.tensorflow.org/guide/keras/overview) API.
For more examples of using Keras, check out the [tutorials](https://www.tensorflow.org/tutorials/keras/). To learn more about building models with Keras, read the [guides](https://www.tensorflow.org/guide/keras). If you want learn more about loading and preparing data, see the tutorials on [image data loading](https://www.tensorflow.org/tutorials/load_data/images) or [CSV data loading](https://www.tensorflow.org/tutorials/load_data/csv).
# CNN: NN with image processing
# Introduction to Data Augmentation
Data augmentation is a technique used in machine learning and deep learning to increase the size of a dataset by creating new versions of the data. This can be useful in situations where the original dataset is small or when more variation in the data is needed to improve the model's accuracy. Data augmentation can also help prevent overfitting by exposing the model to a wider range of inputs.
There are many types of data augmentation techniques, but some common ones include:
1. Flipping and Rotating: In image classification tasks, flipping and rotating the images can help increase the variety of the data. This can be done horizontally, vertically, or both, and rotating an image by a small angle can create a new version of the image that is slightly different from the original.
2. Scaling and Cropping: Resizing an image to a different size or cropping out a portion of the image can also create new versions of the data. This can be particularly useful when working with images of different sizes, as it can help ensure that the model is able to recognize the same object at different scales.
3. Adding Noise: Adding random noise to an image can help make the model more robust to variations in the data. This can include things like Gaussian noise, salt-and-pepper noise, or speckle noise.
4. Changing Colors: Adjusting the color balance or hue of an image can create new versions of the data that are different from the original. This can be particularly useful when working with images that have different lighting conditions or color schemes.
5. Translation: Translating an image by a small amount in any direction can create a new version of the data that is slightly different from the original. This can help the model better handle images with objects in different positions.
Now that we've gone over some of the most common data augmentation techniques, let's discuss how to actually implement them in practice.
1. Flipping and Rotating: This can be done using the `ImageDataGenerator` class in the `keras.preprocessing.image` module in Python. Here's an example:
```
pythonCopy code
from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
horizontal_flip=True,
vertical_flip=True,
rotation_range=30
)
```
This code creates an instance of the `ImageDataGenerator` class with the `horizontal_flip`, `vertical_flip`, and `rotation_range` parameters set to True and 30, respectively. These parameters indicate that the images should be randomly flipped horizontally and vertically, and randomly rotated by up to 30 degrees.
2. Scaling and Cropping: This can also be done using the `ImageDataGenerator` class, as shown in the following example:
```
pythonCopy code
from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
zoom_range=0.2,
width_shift_range=0.1,
height_shift_range=0.1,
fill_mode='nearest'
)
```
This code creates an instance of the `ImageDataGenerator` class with the `zoom_range`, `width_shift_range`, and `height_shift_range` parameters set to 0.2, 0.1, and 0.1, respectively. These parameters indicate that the images should be randomly zoomed in or out by up to 20%, and randomly shifted horizontally and vertically by up to 10% of the image's width or height.
3. Adding Noise: This can be done using the `opencv` library in Python. Here's an example:
```
import cv2
import numpy as np
def add_gaussian_noise(image):
row, col, ch = image.shape
mean = 0
var = 0.1
sigma = 0.5
gauss = np.random.normal(mean, sigma, (row, col, ch))
gauss = gauss.reshape(row, col, ch)
noisy = image + gauss
return noisy
```
This code defines a function called `add_gaussian_noise` that takes an image as input and adds Gaussian noise to it using the `np.random.normal` function.
4. Changing Colors: This can also be done using the `ImageDataGenerator` class, as shown in the following example:
```python
from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
brightness_range=[0.5, 1.5],
channel_shift_range=50,
hue_shift_range=0.2
)
```
This code creates an instance of the `ImageDataGenerator` class with the `brightness_range`, `channel_shift_range`, and `hue_shift_range` parameters set to \[0.5, 1.5], 50, and 0.2, respectively. These parameters indicate that the images should be randomly brightened or darkened by up to 50%, randomly shifted in color channels by up to 50, and randomly shifted in hue by up to 0.2.
5. Translation: This can also be done using the `ImageDataGenerator` class, as shown in the following example:
```python
from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
width_shift_range=0.2,
height_shift_range=0.2
)
```
This code creates an instance of the `ImageDataGenerator` class with the `width_shift_range` and `height_shift_range` parameters set to 0.2. These parameters indicate that the images should be randomly shifted horizontally and vertically by up to 20% of the image's width or height.
Once you've defined the data augmentation techniques you want to use, you can apply them to your dataset by calling the `flow_from_directory` method on your `ImageDataGenerator` object, as shown in the following example:
```python
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rotation_range=30,
width_shift_range=0.2,
height_shift_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
vertical_flip=True,
fill_mode='nearest'
)
train_generator = train_datagen.flow_from_directory(
'path/to/training/directory',
target_size=(224, 224),
batch_size=32,
class_mode='categorical'
)
```
This code creates an instance of the `ImageDataGenerator` class with the data augmentation techniques we defined earlier, and then creates a generator object using the `flow_from_directory` method. This generator object can be used to train a model using the `fit_generator` method, like so:
```python
model.fit_generator(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=10
)
```
# Brief introduction to miscellaneous Neural Networks