Page 1 of 1

Error Derivative in Backpropagation neural network from scracth

Posted: Tue Aug 02, 2022 10:17 pm
by Al.Vioky
First, I created the next forward propagate function
First, I created the next forward propagate function

Code: Select all

# save activations per layer
activations = []
for i in range(len(layers)):
   a = np.zeros(layers[i])
   activations.append(a)
self.activations = activations

def forward_propagate(self, inputs):
   # the input layer activation is just the input itself
   activations = inputs

   # save the activations for backpropogation
   self.activations[0] = activations

   # iterate through the network layers
   for b, w in zip(self.bias,self.weights):
      for i in range(len(self.pesos)-1):
        net_inputs = self._sigmoid(np.dot(activations[i-1],w)+b)
        self.activations[i+1]=net_inputs
   return activaciones

After that, I'd like to train the newtork so I design the next funcion

Code: Select all

def train(self, inputs, targets, epochs, learning_rate):
   # now enter the training loop
   for i in range(epochs):
      sum_errors = 0

      # iterate through all the training data
      for j, input in enumerate(inputs):
         target = targets[j]

         # activate the network!
         output = self.forward_propagate(input)

         error = (2*(target - output))/len(output) #derivative of MSE

         self.back_propagate(error)

Finally. I created the backpropagate function that received the error as an input, where the error es the derivative of the MSE, after that the first derivate I wanna calculate is dE/dW_[i] as you can see bellow

Code: Select all

dW=[]
db=[]
for i in range(len(layers)-1):
   derW=np.zeros((layers[i], layers[i+1]))
   derb=np.zeros((layers[i+1])).reshape(1, layers[i+1])
   dW.append(derW) 
   db.append(derb)
self.dW=dW 
self.db=db

def back_propagate(self, error):
    #dE/dW_i=2(y-a_[i+1])/m*s'(h_[i+1])a_[i] donnde (y-a_[i+1]) derivative of MSE
    #s'(h_[i+1])=s(h_[i+1])*(1-s(h_[i+1])) derivative of sigmoid activacion function
    #s(h_[i+1])=a_[i+1]
    #delta=2(y-a_[i+1])/m*s'(h_[i+1])
    #dE/dW_[i-1]=(y-a_[i+1])s'(h_[i+1])*W_[i]*s'(h_[i])a_[i]
    for i in reversed(range(len(self.dW))):
      # get activation for previous layer
      activations = self.activations[i+1]

      # apply sigmoid derivative function
      delta = error*self._sigmoid_derivative(activations)

      # reshape delta as to have it as a 2d array
      delta_re=delta.reshape(delta.shape[0], -1).T

      # get activations for current layer
      current_activations = self.activations[i]

      # reshape activations as to have them as a 2d column matrix
      current_activations = current_activations.reshape(current_activations.shape[0],-1)

      # save derivative after applying matrix multiplication
      self.dW[i] = np.dot(current_activations, delta_re)

      # backpropogate the next error
      error = np.dot(delta, self.weights[i].T)


but when I run the code with the next data

Code: Select all

# create a dataset to train a network for the sum operation
items = np.array([[random()/2 for _ in range(2)] for _ in range(1000)])
targets = np.array([[i[0] + i[1]] for i in items])

# create a Multilayer Perceptron with one hidden layer
mlp = MLP(2, [5], 1)

# train network
mlp.train(items, targets, 50, 0.1)

I have the next error

Code: Select all

---> 69       error = np.dot(delta, self.weights[i].T)
shapes (2,) and (1,5) not aligned: 2 (dim 0) != 1 (dim 0)

I understand the error but not how to fix it. Any help?