Method and system for training a neural network with adaptive weight updating and adaptive pruning in principal component space

Claim 2

A neural network having a plurality of weights for receiving a sequence of signal inputs x t ,x t+1 , x t+2 . . . , each input x t comprising n signal components x 1 (t), x 2 (t-1), . . . , x n (t-(n-1)) and for generating an output signal that anticipates the behavior of said input signal for a number of time samples ahead, said neural network comprising: transformation means for transforming a set of n signal inputs into a set of principal components having a saliency assigned to each of said principal component; pruning means, coupled to said transformation means, for pruning a number of said principal components that correspond to the smallest saliencies, where the number of said principal components is limited by a sum of said saliencies of said pruned principal components to be less than or equal to a predefined threshold level, leaving a remaining set of principal components; first computing means, coupled to said pruning means, for computing the output signal using said set of remaining principal components: and wherein said neural network an updating means, coupled to said first computing means, for updating the weights of the neural network adaptively based on an error between a target output and the output signal.

Claim 3

A neural network having a plurality of weights for receiving a sequence of signal inputs x t ,x t+1 , x t+2 . . . , each input x t comprising n signal components x 1 (t), x 2 (t-1), . . . , x n (t-(n-1)) and for generating an output signal that anticipates the behavior of said input signal for a number of time samples ahead, said neural network comprising: transformation means for transforming a set of n signal inputs into a set of principal components having a saliency assigned to each of said principal component: pruning means coupled to said transformation means, for pruning a number of said principal components that correspond to the smallest saliencies, where the number of said principal components is limited by a sum of said saliencies of said pruned principal components to be less than or equal to a predefined threshold level, leaving a remaining set of principal components; first computing means, coupled to said pruning means, for computing the output signal using said set of remaining principal components: and updating means, coupled to said first computing means, for updating the weights of the neural network adaptively based on an error between a target output and the output signal, wherein said transformation means includes an estimation means for recursively estimating a current set of principal components from a set of principal components of a previously transformed set of n signal inputs.

Claim 4

A neural network having a plurality of weights for receiving a sequence of signal inputs x t ,x t+1 , x t+2 . . . , each input x t comprising n signal components x 1 (t), x 2 (t-1), . . . , x n (t-(n-1)) and for generating an output signal that anticipates the behavior of said input signal for a number of time samples ahead, said neural network comprising: transformation means for transforming a set of n signal inputs into a set of principal components having a saliency assigned to each of said principal component: pruning means coupled to said transformation means, for pruning a number of said principal components that correspond to the smallest saliencies, where the number of said principal components is limited by a sum of said saliencies of said pruned principal components to be less than or equal to a predefined threshold level, leaving a remaining set of principal components; first computing means, coupled to said pruning means, for computing the output signal using said set of remaining principal components: and updating means, coupled to said first computing means, for updating the weights of the neural network adaptively based on an error between a target output and the output signal, wherein said transformation means includes an estimation means for recursively estimating a current set of principal components from a set of principal components of a previously transformed set of n signal inputs.

Claim 5

The neural network of claim 2, wherein said estimation means estimates said current set of principal components in accordance to the formula R.sub.t =μR.sub.t-1 +(1-μ)x.sub.t x.sub.t.sup.T where R t-1 is a covariance matrix of a previous set of n signal inputs, R t is a covariance matrix of said current set of n signal inputs, μ is a predetermined constant, x t is a current input signal, and x t T is a transpose of said current input signal.

Claim 6

The neural network of claim 2, wherein said estimation means estimates said current set of principal components by directly calculating a matrix Q t and a matrix Λ t , where Q t is a matrix of eigenvectors and Λ t is a matrix of eigenvalues.

Claim 7

The neural network of claim 2, wherein said saliencies are calculated in accordance to the formula s.sub.i (t)= v.sub.i x.sub.i !.sup.T v.sub.i x.sub.i !=v.sub.t.sup.T v.sub.i x.sub.i.sub.2 where x t is the Karhunen-Loeve expansion of x t and v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !.

Claim 8

The neural network of claim 2, wherein said saliencies are calculated in accordance to the formula s.sub.i (t)=v.sub.t.sup.T v.sub.i x.sub.i.sup.2 where x t is the Karhunen-Loeve expansion of x t , v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !, x i 2 is defined as x i 2 (t)=μx i 2 (t-1)+(1-μ)x i 2 (t) and μ is a forgetting factor.

Claim 9

The neural network of claim 2, wherein said saliencies are calculated in accordance to the formula s.sub.i =λ.sub.i v.sub.i.sup.T v.sub.i, where λ i is the ith element on the diagonal of Λ t and v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !.

Claim 10

The neural network of claim 2, wherein said pruning means includes: second computing means for computing an output in principal component space; identifying means, coupled to said second computing means, for identifying said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level; and third computing means, coupled to said identifying means, for computing a pruning vector from said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level.

Claim 11

The neural network of claim 2, wherein said pruning means includes: identifying means for identifying said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level; and fourth computing means, coupled to said identifying means, for computing a weight matrix in regular space from said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level.

Claim 16

The method of claim 11, further comprising the step of: (f) repeating steps (a)-(d) for each new signal input.

Claim 17

The method of claim 11, wherein said estimation step estimates said current set of principal components in accordance to the formula R.sub.t =μR.sub.t-1 +(1-μ)x.sub.t x.sub.t.sup.T where R t-1 is a covariance matrix of a previous set of n signal inputs, R t is a covariance matrix of said current set of n signal inputs, μ is a predetermined constant, x t is a current input signal, and x t T is a transpose of said current input signal.

Claim 18

The method of claim 11 wherein said estimating step estimates said current set of principal components by directly calculating a matrix Q t and a matrix Λ t , where Q t is a matrix of eigenvectors and Λ t is a matrix of eigenvalues.

Claim 19

The method of claim 11, wherein said saliencies are calculated in accordance to the formula s.sub.i (t)= v.sub.i x.sub.i !.sup.T v.sub.i x.sub.i !=v.sub.t.sup.T v.sub.i x.sub.i.sup.2 where x t is the Karhunen-Loeve expansion of x t and v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !.

Claim 20

The method of claim 11, wherein said saliencies are calculated in accordance to the formula s.sub.i (t)=v.sub.t.sup.T v.sub.i x.sub.i.sup.2 where xx t is the Karhunen-Loeve expansion of x t , v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !, x i 2 is defined as x i 2 (t)=μx i 2 (t-1)+(1-μ)x i 2 (t) and μ is a forgetting factor.

Claim 21

The method of claim 11, wherein said saliencies are calculated in accordance to the formula s.sub.i =λ.sub.i v.sub.i.sup.T v.sub.i, where λ i is the ith element on the diagonal of Λ t and v i is a p×1 vector of W t T defined as W t T = v 1 ,v 2 , . . . , v n !.

Claim 22

The method of claim 11, wherein said pruning step includes the steps of: computing an output in principal component space; identifying said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level; and computing a pruning vector from said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level.

Claim 23

The method of claim 11, wherein said pruning step includes the steps of: identifying said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level; and computing a weight matrix in regular space from said principal components that correspond to the smallest saliencies, where a sum of said smallest saliencies is less than a predefined threshold level.

Claim 24

A signal processing system having a neural network for receiving a sequence of signal inputs x t , x t+1 , x t+2 . . . , each input x t comprising n signal components x 1 (t), x 2 (t-1), . . . , x n (t-(n-1)) and generating an output signal that anticipates the behavior of said input signal for a number of time samples ahead, said neural network having a plurality of hierarchically connected nodes forming a plurality of layers, each of said layer consisting of at least one node, said nodes being inter-connected with a plurality of weights, said signal processing system comprising: transformation means for transforming a set of n signal inputs into a set of principal components having a saliency assigned to each of said principal component; pruning means, coupled to said transformation means, for pruning a number of said principal components that correspond to the smallest saliencies, where the number of said pruned principal components is limited by a sum of said saliencies of said pruned principal components to be less than or equal to a predefined threshold level, leaving a remaining set of principal components; computing means, coupled to said pruning means, for computing the output signal of a layer of the neural network using said set of remaining principal components; and updating means, coupled to said computing means, for updating the weights of the neural network adaptively based on an error between a target output and the output signal.

Claim 25

A signal processing system having a neural network for receiving a sequence of signal inputs x t , x t+1 , x t+2 . . . , each input x t comprising n signal components x 1 (t), x 2 (t-1), . . . , x n (t-(n-1)) and generating an output signal that anticipates the behavior of said input signal for a number of time samples ahead, said neural network having a plurality of hierarchically connected nodes forming a plurality of layers, each of said layer consisting of at least one node, said nodes being inter-connected with a plurality of weights, said signal processing system comprising: transformation means for transforming a set of n signal inputs into a set of principal components having a saliency assigned to each of said principal component; pruning means, coupled to said transformation means, for pruning a number of said principal components that correspond to the smallest saliencies, where the number of said pruned principal components is limited by a sum of said saliencies of said pruned principal components to be less than or equal to a predefined threshold level, leaving a remaining set of principal components; computing means, coupled to said pruning means, for computing the output signal of a layer of the neural network using said set of remaining principal components; and updating means, coupled to said computing means, for updating the weights of the neural network adaptively based on an error between a target output and the output signal.

Claim 26

The signal processing system of claim 20, wherein said transformation means includes an estimation means for recursively estimating a current set of principal components from a set of principal components of a previously transformed set of n signal inputs.