Training a neural network using differential input

Claim 2

In an information processing system, a method for training a neural network, said neural network comprising a plurality of interconnected nodes having inputs and outputs, said inputs and outputs including a plurality of network inputs and a network output, said neural network including at least one network parameter, the method comprising the steps of: (a) accepting training data; (b) deriving from said training data a change in network input values between two data points from said training data; (c) deriving from said training data a corresponding desired change in network output value between said two data points; (d) computing an actual change in network output value using said change in network input values, said network including at least one node producing a product of at least two inputs or a product of an input multiplied by itself; (e) computing a difference-error, said difference-error being a difference between said desired change in network output and said actual change in network output; (f) computing an adjustment value for one of said network parameters, said adjustment value including a difference-error-applied, said difference-error-applied derived at least in part from said difference-error; and (g) adjusting said one network parameter by an amount derived at least in part from said adjustment value.

Claim 3

The method according to claim 1 further comprising the step of: (h) computing a derivative of change in network output value with respect to said one parameter; wherein step (f) of computing said adjustment value comprises computing said adjustment value to include said derivative.

Claim 4

The method according to claim 2 wherein step (h) of computing said derivative comprises computing said derivative in an order from an output layer of nodes within said neural network to an input layer of nodes within said neural network.

Claim 5

The method according to claim 3 wherein step (h) of computing said derivative comprises using the chain rule to compute a closed-form expression for said derivative.

Claim 6

The method according to claim 1 wherein step (f) of computing said adjustment value comprises setting said difference-error-applied to said difference-error for computing said adjustment value.

Claim 7

the method according to claim 1 further comprising the step of: (i) computing said difference-error-applied from said difference-error to compensate for coupling between multiple pairs of training points according to a solution to a linear equation, in which linear equation the difference-error of multiple data points are the product of coupling coefficients and difference-error-applied of said multiple pairs of data points.

Claim 8

The method according to claim 1, wherein: said two training data points are henceforth referred to as the first data point-pair; and the step (f) includes computing a coupling from said first data point-pair to a second data point-pair in deriving said difference-error-applied from said difference-error.

Claim 9

The method according to claim 1 wherein step (d) of computing an actual change in network output value comprises running a training network using said change in network input values, said training network having a same topology as said neural network but with an output that is a change in network output.

Claim 10

The method according to claim 1 wherein said training data comprises values for said network inputs and characterization of desired network output values corresponding to said network input values.

Claim 11

A neural network produced according to the training method of claim 1.

Claim 12

In an information processing system, a method for training a neural network, said neural network comprising a plurality of interconnected nodes, each node having at least one input and at least one output, said network having network inputs and a network output, said neural network including at least one network parameter, the method comprising the steps of: (a) constructing a training network from said neural network, said training network accepting differential inputs and producing a differential output; (b) accepting training data; (c) deriving differential input values from said training data; (d) deriving a desired differential output value corresponding to said differential input values from said training data; (e) running said training network using said differential input values to compute an actual differential output value, said network including at least one node producing a product of at least two inputs or a product of an input multiplied by itself; (f) computing a differential-error, said differential-error being a difference between said desired differential output and said actual differential output; (g) computing an adjustment value for one of said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error; and (h) adjusting said one network parameter by an amount derived at least in part from said adjustment value.

Claim 13

In an information processing system, a method for training a neural network, said neural network comprising a plurality of interconnected nodes, each node having at least one input and at least one output, said network having network inputs and a network output, said neural network including at least one network parameter, the method comprising the steps of: (a) constructing a training network from said neural network, said training network accepting differential inputs and producing a differential output; (b) accepting training data; (c) deriving differential input values from said training data; (d) deriving a desired differential output value corresponding to said differential input values from said training data; (e) running said training network using said differential input values to compute an actual differential output value, said network including at least one node producing a product of at least two inputs or a product of an input multiplied by itself; (f) computing a differential-error, said differential-error being a difference between said desired differential output and said actual differential output; (g) computing an adjustment value for one of said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error; and (h) adjusting said one network parameter by an amount derived at least in part from said adjustment value.

Claim 14

The method according to claim 11 wherein: said differential input values are a change in network input values between two data points from said training data; said differential output values, desired and actual, are change in network output value, desired and actual, respectively, between said two data points, whereby said differential-error is a difference-error, which difference-error is an error between said desired change in network output value and said actual change in network output value; said differential-error-applied is a difference-error-applied, said difference-error-applied derived at least in part from said difference-error.

Claim 15

The method according to claim 11 wherein: said differential input values are a slope of network input values; said differential output values, desired and actual, are slope of network output value, desired and actual, respectively, whereby said differential-error is a slope-error, which slope-error is an error between said desired slope of network output value and said actual slope of network output value; said differential-error-applied is a slope-error-applied, said slope-error-applied derived at least in part from said slope-error.

Claim 16

The method according to claim 11 further comprising the step of: (i) computing a derivative of differential network output value with respect to said one parameter; wherein step (g) of computing said adjustment value comprises computing said adjustment value to include said derivative.

Claim 17

A computer program product, for a computer system including a processor and a memory, the computer program product for training a neural network using a training network, said neural network comprising a plurality of interconnected nodes having inputs including at least one network input and outputs including a network output, at least one of said nodes realizing a function of its inputs at an output, said function comprising at least one function parameter, the computer program product comprising: a computer readable storage medium comprising: (a) code that directs the processor to accept training data; (b) code that directs the processor to derive differential input values from said training data; (c) code that directs the processor to derive a desired differential output value corresponding to said differential input values from said training data; (d) code that directs the processor to run said training network using said differential input values to compute an actual differential output value, including code that produces a product of at least two inputs in at least one of said nodes; (e) code that directs the processor to compute an differential-error, said differential-error being a difference between said desired differential output and said actual differential output; (f) code that directs the processor to compute an adjustment value for one of said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error; and (g) code that directs the processor to adjust said one network parameter by an amount derived at least in part from said adjustment value.

Claim 18

A computer program product, for a computer system including a processor and a memory, the computer program product for training a neural network using a training network, said neural network comprising a plurality of interconnected nodes having inputs including at least one network input and outputs including a network output, at least one of said nodes realizing a function of its inputs at an output, said function comprising at least one function parameter, the computer program product comprising: a computer readable storage medium comprising: (a) code that directs the processor to accept training data; (b) code that directs the processor to derive differential input values from said training data; (c) code that directs the processor to derive a desired differential output value corresponding to said differential input values from said training data; (d) code that directs the processor to run said training network using said differential input values to compute an actual differential output value, including code that produces a product of at least two inputs in at least one of said nodes; (e) code that directs the processor to compute an differential-error, said differential-error being a difference between said desired differential output and said actual differential output; (f) code that directs the processor to compute an adjustment value for one of said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error; and (g) code that directs the processor to adjust said one network parameter by an amount derived at least in part from said adjustment value.

Claim 19

A computerized system for training a neural network, said neural network comprising a plurality of interconnected nodes, each node having at least one input and at least one output, said neural network having at least one network input and one network output, said neural network comprising function parameters, the system comprising: (a) an input stage for accepting training data, said input stage configured to derive differential input values and a desired differential output value from said training data; (b) a network evaluator configured to run a training network corresponding to said neural network, using said differential input values, in obtaining a differential-error based on said desired differential output value, said neural network including at least one node producing a product of at least two inputs or the product of one input multiplied by itself; (c) a parameter updater configured to compute an adjustment value for a parameter from said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error, said parameter updater also configured to adjust said network parameter according to said adjustment value.

Claim 20

A computerized system for training a neural network, said neural network comprising a plurality of interconnected nodes, each node having at least one input and at least one output, said neural network having at least one network input and one network output, said neural network comprising function parameters, the system comprising: (a) an input stage for accepting training data, said input stage configured to derive differential input values and a desired differential output value from said training data; (b) a network evaluator configured to run a training network corresponding to said neural network, using said differential input values, in obtaining a differential-error based on said desired differential output value, said neural network including at least one node producing a product of at least two inputs or the product of one input multiplied by itself; (c) a parameter updater configured to compute an adjustment value for a parameter from said network parameters, said adjustment value including a differential-error-applied, said differential-error-applied derived at least in part from said differential-error, said parameter updater also configured to adjust said network parameter according to said adjustment value.

Claim 21

The system according to claim 16 wherein: said parameter updater includes a coupling calculator for computing a coupling from a first set of said differential input values and to a second set of differential input values; and said parameter updater is configured to derive said differential-error-applied from said differential-error using said coupling to thereby compensate for said coupling.