Coursera Deep Learning Course 1 Week 4 notes: Deep neural networks

2017-10-16

• notes
• deep learning

---

Deep Neural Network

Deep L-layer neural network

The word ‘deep’ refers to the number of layer of a neural network (not count the input layer).

Some notations:

• $L$ = number of layer
• $n^{[l]}$ = number of units in layer l ($n_{X} = n^{[0]})$
• $a^{[l]}$ = activations in layer l ($X = a^{[0]}, ~\hat{y} = a^{[L]}, ~a^{[l]} = g^{[l]}(z^{[l]})$).

Forward Propagation in a Deep Network

$$z^{[l]} = w^{[l]}a^{[l-1]} + b^{[l]}$$ $$a^{[l]} = g^{[l]}(z^{[l]})$$

Vectorization for whole training set (stack lowercase matrices in column to obtain capital matrices):

$$Z^{[l]} = W^{[l]}A^{[l-1]} + B^{[l]}$$ $$A^{[l]} = g^{[l]}(Z^{[l]})$$

We can’t avoid having a for loop iterating over all layers.

Getting your matrix dimensions right

No notes.

Why deep representing

No notes.

Building blocks of deep neural networks

---

Figure 1. Forward and backward implementation. Source: Coursera Deep Learning course

---

Forward and Backward Propagation

Forward propagation for layer l

Input $a^{[l - 1]}$

Output $a^{[l]}$, cache($z^{[l]}, ~w^{[l]}, ~b^{[l]}$)

• $z^{[l]} = w^{[l]}a^{[l - 1]} + b^{[l]}$.

• $a^{[l]} = g^{[l]}(z^{[l]})$.

Backward propagation for layer l

Input $da^{[l]}$

Output $da^{[l - 1]}, ~dW^{[l]}, ~db^{[l]}$

• $dz^{[l]} = da^{[l]} * {g^{[l]}}'(z^{[l]})$.

• $dW^{[l]} = dz^{[l]}a^{[l - 1]}$.

• $db^{[l]} = dz^{[l]}$.

• $da^{[l - 1]} = W^{[l]}dz^{[l]}$.

Parameters vs Hyperameters

Parameters: $W^{[l]}, ~b^{[l]}$.

Hyperparameters: control parameters.

• Learning rate $\alpha$.
• Number of iterations.
• Number of hidden layers L.
• Number of hidden units.
• Choice of activation functions.
• Many more…

Therefore, applying deep learning is a very empirical process.

What does this have to do with the brain?

« Coursera Deep Learning Course 1 Week 3 notes: Shallow neural networks Coursera Deep Learning Course 2 Week 1 notes: Practical aspects of Deep Learning »