Optimal filtering by recurrent neural networks

Claim 3

The method of claim 1 wherein said step of selecting a recurrent neural network paradigm and said step of selecting an estimation error criterion are accomplished such that a candidate primary filter of said recurrent neural network paradigm exists that approximates an optimal filter in performance with respect to said estimation error criterion to any predetermined level of accuracy.

Claim 4

The method of claim 2 wherein said step of selecting a recurrent neural network paradigm is accomplished by selecting said recurrent neural network paradigm from the group consisting of multilayer perceptrons with interconnected neurons and multilayer perceptrons with output feedbacks, in each of which the activation function is a bounded, nondecreasing and nonconstant function, and wherein said step of selecting an estimation error criterion is accomplished by selecting said estimation error criterion from the group consisting of the mean square error, the mean absolute deviation, and the mean Huber's error.

Claim 5

The method of claim 1 wherein said step of selecting a recurrent neural network paradigm is accomplished by selecting from the group consisting of multilayer perceptrons with only free output feedbacks.

Claim 6

The method of claim 1 wherein said step of selecting a recurrent neural network paradigm is accomplished by selecting said recurrent neural network paradigm such that the activation function of each outward output node in said recurrent neural network reflects the range of said outward output node's target values.

Claim 7

The method of claim 1 wherein said step of synthesizing said training data is accomplished through minimizing said training criterion by the variation of the weights and initial dynamic state of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 8

The method of claim 1 wherein said step of synthesizing said training data is accomplished through setting the initial dynamic state equal to a canonical initial dynamic state and minimizing said training criterion by the variation of the weights of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 9

The method of claim 1 wherein said step of synthesizing said training data is accomplished through a minimization algorithm, in which the derivatives of said training criterion are calculated by a differentiation technique selected from the group consisting of those differentiation techniques of backpropagation through time and real time recurrent learning, and in which adjustment of the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks is performed iteratively by a technique selected from the group consisting of the conjugate gradient methods, the quasi-Newton methods, and the gradient descent methods.

Claim 10

The method of claim 1 wherein said step of synthesizing said training data is accomplished through applying the extended Kalman filtering in an iterative manner to estimate the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 11

The method of claim 1 wherein said step of synthesizing said training data is through using a global optimization method selected from the group consisting of annealing methods and clustering methods.

Claim 12

The method of claim 1 wherein said step of implementing said primary filter is accomplished by a method selected from the group consisting of programming a computer, programming a microprocessor, programming an analog device, programming a digital device, copying the weights and initial dynamic state into a neural network chip, downloading the weights and initial dynamic state into a neural network chip, making an electronic circuit, making a mechanical device, constructing an optical apparatus, and constructing an electro-optical apparatus.

Claim 13

The method of claim 1 wherein said step of synthesizing said training data is performed directly on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, thereby completing said step of implementing said primary filter at the same time.

Claim 14

The method of claim 1 wherein said step of implementing said primary filter is accomplished by implementing said primary filter on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, and wherein said step of synthesizing said training data into a primary filter is accomplished through including said piece of hardware in the process of training each of a collection of recurrent neural networks of said recurrent neural network paradigm so as to overcome the anomalies and imperfections of said piece of hardware.

Claim 21

The method of claim 16 wherein said step of selecting a recurrent neural network paradigm and said step of selecting an estimation error criterion are accomplished such that a candidate primary filter with augmented exogenous input nodes of said recurrent neural network paradigm exists that approximates an optimal filter in performance with respect to said estimation error criterion to any predetermined level of accuracy.

Claim 24

The method of claim 16 wherein said step of selecting a recurrent neural network paradigm is accomplished by selecting from the group consisting of multilayer perceptrons with only free output feedbacks.

Claim 25

The method of claim 16 wherein said step of selecting said recurrent neural network paradigm is accomplished by selecting said recurrent neural network paradigm in which the activation function of each outward output node reflects the range of said outward output node's target values.

Claim 26

The method of claim 16 wherein said step of synthesizing said augmented training data is accomplished through minimizing said training criterion by the variation of the weights and initial dynamic state of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 27

The method of claim 16 wherein said step of synthesizing said augmented training data is accomplished through setting the initial dynamic state equal to a canonical initial dynamic state and minimizing said training criterion by the variation of the weights of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 28

The method of claim 16 wherein said step of synthesizing said augmented training data is accomplished through a minimization algorithm, in which the derivatives of said training criterion are calculated by a differentiation technique selected from the group consisting of those differentiation techniques of backpropagation through time and real time recurrent learning, and in which adjustment of the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks is performed iteratively by a technique selected from the group consisting of the conjugate gradient methods, the quasi-Newton methods, and the gradient descent methods.

Claim 29

The method of claim 16 wherein said step of synthesizing said augmented training data is accomplished through applying extended Kalman filtering in an iterative manner to estimate the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 30

The method of claim 16 wherein said step of synthesizing said augmented training data is through using a global optimization method selected from the group consisting of annealing methods and clustering methods.

Claim 31

The method of claim 16 wherein said step of implementing said primary filter with augmented exogenous input nodes is accomplished by a method selected from the group consisting of programming a computer, programming a microprocessor, programming an analog device, programming a digital device, copying the weights and initial dynamic state into a neural network chip, downloading the weights and initial dynamic state into a neural network chip, making an electronic circuit, making a mechanical device, constructing an optical apparatus, and constructing an electro-optical apparatus.

Claim 32

The method of claim 16 where in said step of synthesizing said augmented training data is performed directly on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, thereby completing said step of implementing said primary filter with augmented exogenous input nodes at the same time.

Claim 33

The method of claim 16 wherein said step of implementing said primary filter with augmented exogenous input nodes is accomplished by implementing said primary filter with augmented exogenous input nodes on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, and wherein said step of synthesizing said augmented training data into a primary filter with augmented exogenous input nodes is accomplished through including said piece of hardware in the process of training each of a collection of recurrent neural networks of said recurrent neural network paradigm so as to overcome the anomalies and imperfections of said piece of hardware.

Claim 38

A primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process with respect to a selected estimation error criterion, said primary filter being an implementation of a recurrent neural network of a selected recurrent neural network paradigm, and said primary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing an estimate of one signal vector of said n-dimensional vector-valued signal process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 39

A primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process with respect to a selected estimation error criterion, said primary filter being an implementation of a recurrent neural network of a selected recurrent neural network paradigm, and said primary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing an estimate of one signal vector of said n-dimensional vector-valued signal process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 40

The primary filter of claim 31 wherein said recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing, with respect to a training criterion constructed on the basis of said estimation error criterion, training data, comprising realizations of said signal process and corresponding realizations of said measurement process, into said recurrent neural network.

Claim 41

The primary filter of claim 32 wherein said recurrent neural network paradigm is such that a candidate recurrent neural network of said recurrent neural network paradigm exists that approximates an optimal filter in performance with respect to said estimation error criterion to any predetermined level of accuracy.

Claim 42

The primary filter of claim 33 wherein said recurrent neural network paradigm is selected from the group consisting of multilayer perceptrons with interconnected neurons and multilayer perceptrons with output feedbacks, in each of which the activation function is a bounded, nondecreasing and nonconstant function, and wherein said estimation error criterion is selected from the group consisting of the means square error, the mean absolute deviation, and the mean Huber's error.

Claim 43

The primary filter of claim 32 wherein said recurrent neural network paradigm is selected from the group consisting of multilayer perceptrons with free output feedbacks.

Claim 44

The primary filter of claim 32 wherein the activation function of each outward output node in said recurrent neural network reflects the range of said outward output node's target values.

Claim 45

The primary filter of claim 32 wherein said training data is synthesized through minimizing said training criterion by the variation of the weights and initial dynamic state of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 46

The primary filter of claim 32 wherein said training data is synthesized through setting the initial dynamic state equal to a canonical initial dynamic state and minimizing said training criterion by the variation of the weights of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 47

The primary filter of claim 32 wherein said training data is synthesized through using a minimization algorithm, in which the derivatives of said training criterion are calculated by a differentiation technique selected from the group consisting of those differentiation techniques of backpropagation through time and real time recurrent learning, and in which adjustment of the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks is performed iteratively by a technique selected from the group consisting of the conjugate gradient methods, the quasi-Newton methods, and the gradient descent methods.

Claim 48

The primary filter of claim 32 wherein said training data is synthesized through applying extended Kalman filtering in an iterative manner to estimate said weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 49

The primary filter of claim 32 wherein said training data is synthesized through using a global optimization method selected from the group consisting of annealing methods and clustering methods.

Claim 50

The primary filter of claim 32 wherein said primary filter is implemented by a procedure selected from the group consisting of programming a computer, programming a microprocessor, programming an analog device, programming a digital device, copying weights and initial dynamic state in a neural network chip, making an electronic circuit, making a mechanical device, constructing an optical apparatus, and constructing an electro-optical apparatus.

Claim 51

The primary filter of claim 32 wherein said training data is synthesized directly on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device and an electro-optical apparatus.

Claim 52

The primary filter of claim 32 wherein said primary filter is an implementation on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, and wherein said training data is synthesized through including said piece of hardware in the process of training each of a collection of recurrent neural networks of said recurrent neural network paradigm so as to overcome the anomalies and imperfections of said piece of hardware.

Claim 53

An ancillary filter for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, said ancillary filter being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 54

An ancillary filter for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, said ancillary filter being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 55

The ancillary filter of claim 45 wherein said ancillary recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing realizations of an ancillary signal process, selected on the basis of said estimation error statistic process, and corresponding realizations of said measurement process into said ancillary recurrent neural network.

Claim 56

An ancillary filter for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, said ancillary filter being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one output vector of said primary filter at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 57

An ancillary filter for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, said ancillary filter being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one output vector of said primary filter at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 58

The ancillary filter of claim 47 wherein said ancillary recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing realizations of an ancillary signal process, selected on the basis of said estimation error statistic process, corresponding realizations of said primary filter's output process, and corresponding realizations of said measurement process into said ancillary recurrent neural network.

Claim 59

A primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process with respect to a selected estimation error criterion, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said primary filter with augmented exogenous input nodes being an implementation of a recurrent neural network paradigm, and said primary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing an estimate of one signal vector of said n-dimensional vector-valued signal process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 60

A primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process with respect to a selected estimation error criterion, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said primary filter with augmented exogenous input nodes being an implementation of a recurrent neural network paradigm, and said primary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing an estimate of one signal vector of said n-dimensional vector-valued signal process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 61

The primary filter with augmented exogenous input nodes of claim 49 wherein said recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing, with respect to a training criterion constructed on the basis of said estimation error criterion, augmented training data, comprising a set of exemplary time functions, and realizations of said signal process and realizations of said measurement process, that correspond to each said exemplary time function, into said recurrent neural network.

Claim 62

The primary filter with augmented exogenous input nodes of claim 50 wherein said recurrent neural network paradigm is such that a candidate primary filter with augmented exogenous input nodes of said recurrent neural network paradigm exists that approximates an optimal filter in performance with respect to said estimation error criterion to any predetermined level of accuracy.

Claim 63

The primary filter with augmented exogenous input nodes of claim 51 wherein said recurrent neural network paradigm is selected from the group consisting of multilayer perceptrons with interconnected neurons and multilayer perceptrons with output feedbacks, in each of which the activation function is a bounded, nondecreasing and nonconstant function, and wherein said estimation error criterion is selected from the group consisting of the mean square error, the mean absolute deviation, and the mean Huber's error.

Claim 64

The primary filter with augmented exogenous input nodes of claim 50 wherein said recurrent neural network paradigm is selected from the group consisting of multilayer perceptrons with free output feedbacks.

Claim 65

The primary filter with augmented exogenous input nodes of claim 50 wherein the activation function of each outward output node in said recurrent neural network reflects the range of said outward output node's target values.

Claim 66

The primary filter with augmented exogenous input nodes of claim 50 wherein said means for synthesizing said augmented training data is through minimizing said training criterion by the variation of the weights and initial dynamic state of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 67

The primary filter with augmented exogenous input nodes of claim 50 wherein said augmented training data is synthesized through setting the initial dynamic state equal to a canonical initial dynamic state and minimizing said training criterion by the variation of the weights of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 68

The primary filter with augmented exogenous input nodes of claim 50 wherein said augmented training data is synthesized through using a minimization algorithm, in which the derivatives of said training criterion are calculated by a differentiation technique selected from the group consisting of those differentiation techniques of backpropagation through time and real time recurrent learning, and in which adjustment of the weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks is performed iteratively by a technique selected from the group consisting of the conjugate gradient methods, the quasi-Newton methods, and the gradient descent methods.

Claim 69

The primary filter with augmented exogenous input nodes of claim 50 wherein said augmented training data is synthesized through applying extended Kalman filtering in an iterative manner to estimate said weights and the initial dynamic state, if said initial dynamic state is not set equal to a canonical initial dynamic state, of each of a collection of recurrent neural networks of said recurrent neural network paradigm, of different sizes, in comparison of size versus accuracy to optimize cost-effectiveness.

Claim 70

The primary filter with augmented exogenous input nodes of claim 50 wherein said augmented training data is synthesized through using a global optimization method selected from the group consisting of annealing methods and clustering methods.

Claim 71

The primary filter with augmented exogenous input nodes of claim 50 wherein said primary filter with augmented exogenous input nodes is implemented by a procedure selected from the group consisting of programming a computer, programming a microprocessor, programming an analog device, programming a digital device, copying weights and initial dynamic state into a neural network chip, making an electronic circuit, making a mechanical device, constructing an optical apparatus, and constructing an electro-optical apparatus.

Claim 72

The primary filter with augmented exogenous input nodes of claim 50 wherein said training data is synthesized directly on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device and electro-optical apparatus.

Claim 73

The primary filter with augmented exogenous input nodes of claim 50 wherein said primary filter with augmented exogenous input nodes is an implementation on a piece of hardware selected from the group consisting of a neural network chip, an electronic circuit, an optical device, and an electro-optical apparatus, and wherein said training data is synthesized through including said piece of hardware in the process of training each of a collection of recurrent neural networks of said recurrent neural network paradigm so as to overcome the anomalies and imperfections of said piece of hardware.

Claim 74

An ancillary filter with augmented exogenous input nodes for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said ancillary filter with augmented exogenous input nodes being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 75

An ancillary filter with augmented exogenous input nodes for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said ancillary filter with augmented exogenous input nodes being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 76

The ancillary filter with augmented exogenous input nodes of claim 63 wherein said ancillary recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing a set of exemplary time functions, corresponding realizations of an ancillary signal process, selected on the basis of said estimation error statistic process, and corresponding realizations of said measurement process into said ancillary recurrent neural network.

Claim 77

An ancillary filter with augmented exogenous input nodes for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said ancillary filter with augmented exogenous input nodes being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process, one output vector of said primary filter and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 78

An ancillary filter with augmented exogenous input nodes for evaluating a q-dimensional vector-valued estimation error statistic process, that is required for a primary filter with augmented exogenous input nodes for processing an m-dimensional vector-valued measurement process to estimate an n-dimensional vector-valued signal process, which signal and measurement processes are time-variant with said signal and measurement processes' time-variant property described by a p-dimensional vector-valued time function, said ancillary filter with augmented exogenous input nodes being an implementation of an ancillary recurrent neural network of a selected ancillary recurrent neural network paradigm, and said ancillary filter with augmented exogenous input nodes comprising (1) a plurality of neurons, each neuron being means for receiving said neuron's input values, processing said input values and producing an output value, that is called the activation level of said neuron; (2) a plurality of connections, each connection being means for multiplying a first neuron's activation level by said connection's weight and sending the resulting multiple to a second neuron as one of said second neuron's input values; (3) a plurality of feedback connections, each feedback connection being means for holding a third neuron's activation level for one unit of time, multiplying said third neuron's activation level by said feedback connection's weight, and sending the resulting multiple to a fourth neuron as one of said fourth neuron's input values; (4) at least one input terminal, being means for inputing one measurement vector of said m-dimensional vector-valued measurement process, one output vector of said primary filter and one vector of said p-dimensional vector-valued time function at a time, said at least one input terminal being connected to at least one exogenous input node included in said plurality of neurons; and (5) at least one output terminal, being means for outputing one estimation error statistic vector of said q-dimensional vector-valued estimation error statistic process at a time, said at least one output terminal being connected to at least one outward output node included in said plurality of neurons.

Claim 79

The ancillary filter with augmented exogenous input nodes of claim 65 wherein said ancillary recurrent neural network's architecture, weights and initial dynamic state are determined by synthesizing a set of exemplary time functions, corresponding realizations of an ancillary signal process, selected on the basis of said estimation error statistic process, corresponding realizations of the output process of said primary filter with augmented exogenous input nodes, and corresponding realizations of said measurement process into said ancillary recurrent neural network.