Neural network for maximum likelihood classification with supervised and unsupervised training capability

Claim 1

A neural network for classifying input vectors to one of a plurality of outcome classes comprising: an input layer comprising a plurality of input terminals each for receiving a component of an input vector; two hidden layers, including (i) a first hidden layer having "H" first layer nodes, each being connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, wherein the threshold value is determined as t.sub.i =uT L-T s.sub.i, where "u" is a mean vector of training input vectors used to train the neural network during the training operation, "L" is a Choleski factor of a covariance matrix for the training input vectors, "T" and "-T" represent a transpose and inverse transpose operation, respectively, and "s i " represents a vector defining a surface point on an H-component grid defined on a unit sphere having a number of dimensions corresponding to the number of components of an input vector, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector.

Claim 12

A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of: selecting a value for "K" corresponding to the number of second layer nodes; applying a plurality of training input vectors each of known outcome class to the neural network to generate threshold values and weighting values of said first and second layer nodes; determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.

Claim 13

A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of: selecting a value for "K" corresponding to the number of second layer nodes; applying a plurality of training input vectors each of known outcome class to the neural network to generate threshold values and weighting values of said first and second layer nodes; determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.

Claim 14

A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of: selecting a value for "K" corresponding to the second of second layer nodes; applying a plurality of sets of training input vectors each of unknown outcome class, the training input vectors being related within each set by a predetermined parameter, so that each set of training input vectors is a sequence of a hidden Markov model and each outcome class represented by one of said output nodes is a state of the hidden Markov model to the neural network to generate threshold values and weighting values of said first and second layer nodes; determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.

Claim 15

A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of: selecting a value for "K" corresponding to the second of second layer nodes; applying a plurality of sets of training input vectors each of unknown outcome class, the training input vectors being related within each set by a predetermined parameter, so that each set of training input vectors is a sequence of a hidden Markov model and each outcome class represented by one of said output nodes is a state of the hidden Markov model to the neural network to generate threshold values and weighting values of said first and second layer nodes; determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.