Distributed parallel processing network wherein the connection weights are generated using stiff differential equations

Claim 1

In a parallel distributed processing network implemented in hardware alone or in a digital computer operating in accordance with a program, the network being of the type having: a first, input, set of from one to i processing elements, each input processing element having an input port and an output port, each of the i input processing elements being connectible to an input signal at the input port thereof and each being operable to produce an activation signal A1[i] at the output port thereof; a second, intermediate, set of from one to k processing elements, each processing element having an input port and an output port, each of the k intermediate processing elements being operable to produce an activation signal A2[k] at the output port thereof; a third, output, set of from one to j processing elements, each processing element having an input port and an output port, each of the j output processing elements being operable to produce an activation signal A3[j] at the output port thereof; each of the i processing elements in the input set of processing elements being connected to at least one of the k processing elements in the intermediate set of processing elements with each such interconnection having a connection weight W1[i,k] associated therewith, each of the k processing elements in the set of intermediate processing elements being connected to at least one of the j processing elements in the set of output processing elements with each such interconnection having a connection weight W2[k,j] associated therewith, the network being arranged such that a set of p training signals, applied at the input ports of the input processing element produces activation signals A1[i,p], A2[k,p] and A3[j,p], each of the p training signals being associated with a predetermined one of a set of V predetermined desired output signals, wherein the improvement comprises: the activation signals A1, A2, and A3 and the connection weights W1, W2 of the interconnections between the input processing elements and the output processing elements being related by a first set of stiff differential equations ##EQU7## the activation signals A2 and A3 and the connection weights W2 of interconnections between the intermediate processing elements and the output processing elements are related by a second set of stiff differential equations ##EQU8## and wherein the weights W1 and W2 are, respectively, calculated by converging to the steady state solutions of the first and second sets of stiff differential equations.

Claim 10

A method for training a parallel distributed processing system of the type having a first, input, set of from one to i input processing elements, each input processing element having an input port and an output port, each of the i input processing elements being connectible to an input signal at the input port thereof and each being operable to produce an activation signal A1[i] at the output port thereof; a second, intermediate, set of from one to k intermediate processing elements, each processing element having an input port and an output port, each of the k intermediate processing elements being operable to produce an activation signal A2[k] at the output port thereof; a third, output, set of from one to j processing elements, each processing element having an input port and an output port, each of the j output processing elements being operable to produce an activation signal A3[j] at the output port thereof; each of the i input processing elements being connected to at least one of the k intermediate processing elements with each such interconnection having a connection weight W1[i,k] associated therewith, each of the k intermediate processing elements being connected to at least one of the j output processing elements with each such interconnection having a connection weight W2[k,j] associated therewith, the network being arranged such that a set of p training signals, applied at the input ports of the input processing elements produces activation signals A1[i,p], A2[k,p] and A3[j,p], each of the p training signals being associated with a predetermined one of a set of V predetermined desired output signals, the method comprising the steps of: (a) defining the relationship between the activation signals A1, A2, A3 and the weights W1 and W2 in the form of a first and a second set of stiff differential equations ##EQU9## (b) solving the sets of differential equations to find the values of the weights W1 and W2; and (c) adjusting the weights of the network such that the weights W1 and W2 correspond to the solution sets of the first and second sets of stiff differential equations.

Claim 11

A method for training a parallel distributed processing system of the type having a first, input, set of from one to i input processing elements, each input processing element having an input port and an output port, each of the i input processing elements being connectible to an input signal at the input port thereof and each being operable to produce an activation signal A1[i] at the output port thereof; a second, intermediate, set of from one to k intermediate processing elements, each processing element having an input port and an output port, each of the k intermediate processing elements being operable to produce an activation signal A2[k] at the output port thereof; a third, output, set of from one to j processing elements, each processing element having an input port and an output port, each of the j output processing elements being operable to produce an activation signal A3[j] at the output port thereof; each of the i input processing elements being connected to at least one of the k intermediate processing elements with each such interconnection having a connection weight W1[i,k] associated therewith, each of the k intermediate processing elements being connected to at least one of the j output processing elements with each such interconnection having a connection weight W2[k,j] associated therewith, the network being arranged such that a set of p training signals, applied at the input ports of the input processing elements produces activation signals A1[i,p], A2[k,p] and A3[j,p], each of the p training signals being associated with a predetermined one of a set of V predetermined desired output signals, the method comprising the steps of: (a) defining the relationship between the activation signals A1, A2, A3 and the weights W1 and W2 in the form of a first and a second set of stiff differential equations ##EQU9## (b) solving the sets of differential equations to find the values of the weights W1 and W2; and (c) adjusting the weights of the network such that the weights W1 and W2 correspond to the solution sets of the first and second sets of stiff differential equations.