Part-I: Neural Net Backpropagation using representative equation transformation


Backpropagation in Neural Networks (NN) is a critical step in adjusting the weights of each layer and making the NN model more accurate based on the provided training data.

In previous articles, we derived the backpropagation for a Neural Networks using the architecture of the network. In this series of articles, we convert the neural network architecture into a representative linear equation and use it to derive the backpropagation.

As in other articles, we start with a very simple NN and proceed to more complex NN in subsequent articles. Consider the NN below with 1 hidden layer and only a single neuron in each layer.


We have derived the backpropagation of this NN without using the equation form here.

The output of this NN is \(r_o\), and the prediction error for each training sample \(E\) is defined as:
$$
E = {(t - r_o)^2\over 2} \text { ...eq.(1.1)}
$$
where t is given(expected) output for that specific training example. Our goal is minimize \(E\) by making \(r_o\) as close to t.

In the output layer, \(r_o\) is given as \(r_o=w_o.r_h\). Putting the value of \(r_h\) we can write,\(r_o=w_o.(w_h.i)\). Hence the linear equation:
  \(r_o=w_o.(w_h.i)\)

is the linear representative equation of the NN shown above. For any input i, we will get its corresponding output \(r_o\). Since, i is fixed, when training this equation the only values we can modify is of \(w_o\) and \(w_h\). While we adopt a brute force approach and arbitrarily modify \(w_o\) and \(w_h\), we are not guaranteed to come to optimized values of \(w_o\) and \(w_h\). There is a method to this madness, and that method is backpropagation.

We know that during training, \(E = {(t - r_o)^2\over 2}\). Putting the full value of \(r_o\), we get:
$$
E = {(t - (w_o.(w_h.i)))^2\over 2} \text { ...eq.(1.2)}
$$
This is representative cost function of the NN. Our goal is to reduce E such than in an ideal case it becomes 0 (this ideal case is extremely rare when training NNs). The way we will reduce E, is by changing (increasing or decreasing) the values of \(w_o\) and \(w_h\). Hence, the main thing we need to find out is how E changes (increases/decreases) as \(w_o\) and \(w_h\) are increased (or decreased). This 'rate of change' of E is determined by partial derivatives of E w.r.t. \(w_o\) and \(w_h\). Or, we need to find \({\partial E\over \partial w_{o}}\) and \({\partial E\over \partial w_{h}}\).

Let us start by finding \({\partial E\over \partial w_{o}}\).
$$
\begin{align}
E &= {(t - (w_o.(w_h.i)))^2\over 2} \text { (from eq.(1.2))}\\
{\partial E\over \partial w_{o}}&={\partial {(t - (w_o.(w_h.i)))^2\over 2}\over \partial w_o}\\
&={1 \over 2}.{\partial {(t - (w_o.w_h.i))^2}\over \partial w_{o}} \text{ ... eq.(2.1)}\\
\end{align}
$$

Let us revisit the chain rule:
$$
{dx \over dy}= {dx \over dz}.{dz \over dy}
$$
For a partial derivative:
$$
{\partial x \over \partial y}={\partial x\over \partial z}.{\partial z\over \partial y}
$$
Applying the chain rule to right term of eq.(2.1):
$$
\text{Here, } z = {t - (w_o.w_h.i)},x=z^2, \text{ and } y=w_o \text{ ,therefore, }\\
\begin{align}
{1 \over 2}.{\partial {(t - (w_o.w_h.i))^2}\over \partial w_{o}}&={1 \over 2}.{\partial x\over \partial y}\\
&={1 \over 2}.{\partial x\over \partial z}.{\partial z\over \partial y}\\
&={1 \over 2}.{\partial z^2\over \partial z}.{\partial z\over \partial y}\\
&={1 \over 2}.{\partial (t - (w_o.w_h.i))^2\over \partial (t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{o}}\\
&={1 \over 2}.{2(t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{o}}\\
&={(t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{o}} \text{ ... eq.(2.2)}\\
&={(t - (w_o.w_h.i))}.({\partial t \over \partial w_{o}} - {\partial (w_o.w_h.i)) \over \partial w_{o}})\\
&={(t - (w_o.w_h.i))}.({0} - w_h.i.{\partial (w_o)) \over \partial w_{o}})\\
&={(t - (w_o.w_h.i))}.(-w_h.i) \\
&=-{(t - (w_o.w_h.i))}.(w_h.i) \text{ ... eq.(2.3)}\\
\end{align}
$$
Therefore from eq. (2.3),
$$
\begin{align}
{\partial E\over \partial w_{o}}&=-{(t - (w_o.w_h.i))}.(w_h.i)\text { ... eq.(2.4)}\\
\end{align}
$$
Let us now find \({\partial E\over \partial w_{h}}\).
$$
\begin{align}
E &= {(t - (w_o.(w_h.i)))^2\over 2} \text { (from eq.(1.2))}\\
{\partial E\over \partial w_{h}}&={\partial {(t - (w_o.(w_h.i)))^2\over 2}\over \partial w_h}\\
&={1 \over 2}.{\partial {(t - (w_o.w_h.i))^2}\over \partial w_{h}} \text{ ... eq.(3.1)}\\
\end{align}
$$
From the chain rule we know,
$$
{dx \over dy}= {dx \over dz}.{dz \over dy}
$$
For a partial derivative:
$$
{\partial x \over \partial y}={\partial x\over \partial z}.{\partial z\over \partial y}
$$
Applying the chain rule to right term of eq.(3.1):
$$
\text{Here, } z = {t - (w_o.w_h.i)},x=z^2, \text{ and } y=w_h \text{ ,therefore, }\\
\begin{align}
{1 \over 2}.{\partial {(t - (w_o.w_h.i))^2}\over \partial w_{h}}&={1 \over 2}.{\partial x\over \partial y}\\
&={1 \over 2}.{\partial x\over \partial z}.{\partial z\over \partial y}\\
&={1 \over 2}.{\partial z^2\over \partial z}.{\partial z\over \partial y}\\
&={1 \over 2}.{\partial (t - (w_o.w_h.i))^2\over \partial (t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{h}}\\
&={1 \over 2}.{2(t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{h}}\\
&={(t - (w_o.w_h.i))}.{\partial (t - (w_o.w_h.i))\over \partial w_{h}} \text{ ... eq.(3.2)}\\
&={(t - (w_o.w_h.i))}.({\partial t \over \partial w_{h}} - {\partial (w_o.w_h.i)) \over \partial w_{h}})\\
&={(t - (w_o.w_h.i))}.({0} - w_o.i.{\partial (w_h)) \over \partial w_{h}})\\
&={(t - (w_o.w_h.i))}.(-w_o.i) \\
&=-{(t - (w_o.w_h.i))}.(w_o.i) \text{ ... eq.(3.3)}\\
\end{align}
$$
Therefore from eq. (3.3),
$$
\begin{align}
{\partial E\over \partial w_{h}}&=-{(t - (w_o.w_h.i))}.(w_o.i)\text { ... eq.(3.4)}\\
\end{align}
$$

Once \({\partial E\over \partial w_o}\) and \({\partial E\over \partial w_h}\) are determined, we know if increasing(or decreasing) \(w_o\) and \(w_h\), will reduce E.

The weights for output layer (\(w_o\)) will be updated as:
$$
\begin{align}
\Delta w_o&=-\eta.{\partial E\over \partial w_o}\\
&=-\eta.-{(t - (w_o.w_h.i))}.(w_h.i) \text {from eq.(2.4)}\\
&=\eta.(t - (w_o.w_h.i)).(w_h.i)\\
\Rightarrow w_o &= w_o + \Delta w_o
\end{align}
$$
The weights for hidden layer (\(w_h\)) will be updated as:
$$
\begin{align}
\Delta w_h&=-\eta.{\partial E\over \partial w_h}\\
&=-\eta.-{(t - (w_o.w_h.i))}.(w_o.i) \text {from eq.(3.4)}\\
&=\eta.(t - (w_o.w_h.i)).(w_o.i)\\
\Rightarrow w_h &= w_h + \Delta w_h
\end{align}
$$
Here \(\eta\) is the learning rate and controls the numeric value by which weights are changed. The - sign before the eta is very important. It determines that if E is increasing( \(\partial E\over \partial w_h\) or \(\partial E\over \partial w_o\) is > 0) with an increase in \(w_o\)/\(w_h\) then the value of \(w_o\)/\(w_h\) needs to be decreased, therefore \(\Delta w_h\) and \(\Delta w_o\) will be < 0.


If E is decreasing (\(\partial E\over \partial w_h\)/ \(\partial E\over \partial w_o\) is negative) with an increase in \(w_o\)/\(w_h\) then we continue increasing the value of (w_o\)/\(w_h\), therefore \(\Delta w_h\) and \(\Delta w_o\) will be > 0.

In this article we looked at how a simple NN can be represented by a single linear equation and use that to derive the back propagation. In the next article, we will look at a more complex NN which uses the Sigmoid activation function.


Comments

Post a Comment

Popular posts from this blog

Part III: Backpropagation mechanics for a Convolutional Neural Network

Image
In part-II of this article, we derived the weight update equation for the backpropagation operation of a simple Convolutional Neural Network (CNN). The input for the CNN considered in part-II is a grayscale image, hence, the input is in the form of a single 4x4 matrix. CNNs used in practice however, use color images where each of the Red, Green and Blue (RGB) color spectrums serve as input. Hence, a CNN with the inputs color coded with their respective spectrums look like: In this article we expand upon the original CNN where the input is represented by 3, 4x4 matrices each presenting the pixel intensities for the RGB color spectrum. Hence is we represent each of the pixels with the CNN given is represented by: In this CNN, there are 3, 4x4 input matrices, one 2x2 filter matrix (also known as kernel), a single convolution layer with 1 unit, a single ReLu layer, a single pooling layer (which applies the MaxPool function) and a single fully connected (FC) layer. The elements
Read more

Introducing Convolution Neural Networks with a simple architecture

Image
Convolution Neural Networks (CNNs) are different from standard Neural Networks (NNs) as their input is 2-dimensional vs. 1-dimensional in a standard neural network. While the underlying principles between CNNs and NNs are same, CNNs do introduce some new concepts. In this article, we take a look at how CNNs operate using a very simple CNN architecture, given below. In this simple CNN, there is one 4x4 input matrix, one 2x2 filter matrix (also known as kernel ), a single convolution layer with 1 unit, 1 Rectified Linear Unit (ReLu) layer with one unit, a single pooling layer and a single fully connected (FC) layer. The elements of the filter matrix are equivalent to the unit weights in a standard NN and are updated during the backpropagation phase. Real-life CNNs are significantly more complex than this with several repeating layers of convolutional, ReLu and pooling layers forming a true deep learning network. All the matrix dimensions are also magnitudes higher to represent hig
Read more

Deriving Pythagoras' theorem using Machine Learning

Image
In an earlier article which introduced machine learning, we had seen a comparison of equations derived from scientific principles vs. those derived via machine learning. In this article, we will see how Dythagoras a citizen of the far away planet Darth in the Dilky Day galaxy uses machine learning to derive a correlation between the length of 3 sides of a right-angle triangle. Dythagoras - resemblance to Pythagoras not intentional Unlike humans, the citizens of Darth (Darthians) are not very bright mathematically but are capable of observations and making tools. They had been handed a computer and the Dasmic machine learning library by human astronauts of a SpaceX expedition who had visited them a few months ago. The citizens of Darth need to shorten the travel time between 2 of their biggest cities of Aville and Cville. Currently the only way to goto Cville from Aville is via Bville that takes several hours. Darthians need to know the direct distance between Aville and Cvill
Read more