Neural networks with subdivision

Claim 2

Neural network apparatus for processing input information for a prespecified application to indicate one output category from a set of prespecified output categories for said prespecified application in response to said input information, the input information being supplied to said apparatus as a data array, the neural network apparatus comprising: an input stage for accepting said data array and converting said data array to a corresponding internal representation; a data preprocessor connected to said input stage for analyzing said data array based on a plurality of distinct feature attributes to generate a corresponding plurality of attribute measures; a neural network comprising a plurality of interconnected neurons, said neural network divided into portions, each of said portions being predefined to include a number of said interconnected neurons and said portions being interconnected and exercising a mutual influence, each portion of a first subset of said portions being prespecified with a particular correspondence to one of said feature attributes and adapted to accept from said data preprocessor one of said attribute measures for said data array, said neural network processing said plurality of attribute measures to reach a neural state representative of category attributes, each portion of a second subset of said portions being prespecified with a particular correspondence to one of said category attributes; a data postprocessor connected to said neural network for obtaining category attribute indicators for said category attributes by correlating said neural state with predefined category attribute measures; and an output stage for accepting said category attribute indicators and combining said category attribute indicators in a prespecified manner to generate said output category for said input information.

Claim 3

The neural network apparatus of claim 1 wherein said internal representation is a binary representation.

Claim 4

The neural network apparatus of claim 1 wherein said internal representation is an analog representation.

Claim 5

The neural network apparatus of claim 1 wherein said neural network is an attractor neural network.

Claim 6

The neural network apparatus of claim 1 wherein said neural network further comprises a third subset of portions, each of said third subset of portions not being prespecified with a correspondence to one of said feature attributes or one of said category attributes.

Claim 7

The neural network apparatus of claim 1 wherein said portions of said neural network comprise a prespecified hierarchy of subdivided portions each including a prespecified number of said neurons.

Claim 8

The neural network apparatus of claim 1 wherein said neurons are interconnected via synapses, each synapse characterized by an interconnection weight, and wherein said synapses interconnecting neurons in different portions maintain interconnection weights which are characteristically smaller than interconnection weights maintained by said synapses interconnecting neurons in the same portion.

Claim 9

The neural network apparatus of claim 1 wherein said neurons within a first of said portions are interconnected by S N synapses and where said neurons within a first of said portions are connected to neurons outside of said first of said portions by S synapses, and wherein S is no greater than S N .

Claim 10

The neural network apparatus of claim 1 wherein said input information comprises sensory input information.

Claim 11

The neural network apparatus of claim 1 wherein: said plurality of neurons are interconnected by synapses which are each characterized by an interconnection weight; and said neural network has been trained by a method comprising the steps of: adjusting said interconnection weights a prespecified number of times each comprising the steps of: inputting to said network one of a plurality of prespecified training inputs for processing by said network for a predefined processing interval to produce an output at the end of said processing interval, each training input being associated with a corresponding prespecified training output, sampling said output of said network at the end of said processing interval and comparing said output with said corresponding prespecified training output, and based on said comparison, imprinting corresponding connection weights on said interconnections; and performing a dissociation sequence, comprising the steps of: decreasing said interconnection weights of a prespecified set of said synapses by a first prespecified factor, performing a first relearning procedure a first prespecified number of times each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update role at a first preselected finite temperature below a point of stability for a first preselected number of imprinted memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, increasing said interconnection weights of said prespecified set of synapses by a second prespecified factor, and performing a second relearning procedure on said network a second prespecified number of times each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a second prespecified update rule at a second preselected finite temperature below a point of stability for a second preselected number of imprinted memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, whereby said neural network is trained for said prespecified application.

Claim 12

The neural network apparatus of claim 10 wherein said first and second prespecified factors are each synapse-dependent.

Claim 13

The neural network apparatus of claim 11 wherein said neural network has been trained a prespecified number of times, and wherein said first and second prespecified factors are constant with each said repetition.

Claim 14

The neural network apparatus of claim 10 wherein said prespecified training inputs and outputs correspond to visual patterns.

Claim 15

The neural network apparatus of claim 10 wherein said prespecified training inputs and outputs correspond to acoustic patterns.

Claim 16

The neural network apparatus of claim 10 wherein said prespecified training inputs and outputs correspond to a body of knowledge relating to a particular field of expertise.

Claim 17

The neural network apparatus of claim 1 wherein said plurality of neurons interconnected by synapses which are each characterized by an interconnection weight, said neural network having previously reached a predefined capacity for information storage, and subsequently been trained by a method comprising the steps of: performing a dissociation sequence, comprising the steps of: decreasing said interconnection weights by a first prespecified factor to dissociate said network, performing a first relearning sequence a first prespecified number of times each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update rule at a first preselected finite temperature below a point of stability for a first preselected number of imprinted memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, increasing said interconnection weights by a second prespecified factor to reassociate said network, and performing a second relearning sequence a second prespecified number of times each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update rule at a second preselected finite temperature below a point of stability for a second preselected number of imprinted memories, and imprinting said interconnections with interconnection weights corresponding to said resulting neural state, whereby further information may be imprinted on the network, so that said neural network has achieved additional capacity for information imprinting.

Claim 18

The neural network apparatus of claim 1, wherein: said neural network having previously reached a predefined capacity for information storage, and subsequently been trained by a method comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update rule at a preselected finite temperature below a point of stability for a preselected number of imprint memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, whereby further information may be imprinted on the network, so that said neural network has achieved additional capacity for information imprinting.

Claim 19

The neural network apparatus of claim 1 wherein said input information is a speech pattern and said data array comprises digitized time samples of said speech information; said input stage comprises a signal preprocessor; said data preprocessor is a signal feature processor adapted to apply a plurality of signal attribute functions to said internal representation of said data array at prespecified time intervals; said attribute measures comprise acoustic attribute measures; said category attributes comprise phoneme category attributes; said category attribute measures comprise phoneme category attributes measures; and said output category comprises a speech pattern category output.

Claim 20

The neural network apparatus of claim 18 wherein said speech pattern category output comprises a phoneme.

Claim 21

The neural network apparatus of claim 18 wherein said speech pattern category output comprises a letter.

Claim 22

The neural network apparatus of claim 18 wherein said speech pattern category output comprises a word.

Claim 23

The neural network apparatus of claim 21 wherein said internal representation comprises a signal spectrogram.

Claim 24

The network apparatus of claim 22 wherein said signal attribute functions comprise a measure of signal power integrated over a prespecified time and frequency interval.

Claim 25

The neural network apparatus of claim 1 wherein said output categories are representative of characteristic responses to said input information based on a specific preselected body of knowledge relating to a field of expertise previously presented to said neural network.

Claim 26

The neural network apparatus of claim 18 wherein said sound system comprises a microphone and hardware for digitizing sounds sensed by said microphone.

Claim 27

The neural network apparatus of claim 25 further comprising storage apparatus for recording said speech pattern category output.

Claim 28

The neural network apparatus of claim 25 further comprising display apparatus for displaying said speech pattern category output.

Claim 29

The neural network apparatus of claim 1 wherein: said plurality of neurons are interconnected by synapses which are each characterized by an interconnection weight; and said neural network has been trained by a method comprising the steps of: adjusting said synapse interconnection weights a prespecified number of times each comprising the steps of: inputting to said network one of a plurality of prespecified training inputs for processing by said network for a predefined processing interval to produce an output at the end of said processing interval, each training input being associated with a corresponding prespecified training output, sampling said output of said network at the end of said processing interval and comparing said output with said corresponding prespecified training output, and based on said comparison, imprinting corresponding connection weights on said interconnections; and performing a dissociation sequence, comprising the steps of: decreasing said synapse interconnection weights of a prespecified number of synapses by a first prespecified factor to dissociate said network, performing a first relearning procedure a first prespecified number of limes each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update rule at a first preselected finite temperature below a point of stability for a first preselected number of imprinted memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, increasing said interconnection weights of said prespecified number of synapses by a second prespecified factor to reassociate said network, and performing a second relearning procedure on said reassociated network a second prespecified number of times each comprising the steps of: initiating said network to a random neural state, evolving said neural network from said random neural state to a resulting neural state based on a prespecified update role at a second preselected finite temperature below a point of stability for a second preselected number of imprinted memories, and imprinting said synapses with interconnection weights corresponding to said resulting neural state, whereby said neural network is trained for said prespecified application.

Claim 30

The neural network apparatus of claim 28 wherein said neural network comprises portions which are each predefined with a number of said neurons, and wherein said dissociation sequence further comprises applying said steps of interconnection weight decreasing, relearning, and interconnection weight increasing for each portion of said network.

Claim 31

The neural network apparatus of claim 29 wherein each of said neural network portions comprises a hierarchy of subdivisions each predefined with a number of said neurons, and wherein said dissociation sequence further comprises applying said steps of interconnection weight decreasing, relearning, and interconnection weight increasing for each subdivision of said hierarchy of subdivisions.

Claim 32

The neural network apparatus of claim 28 wherein said dissociation sequence further comprises the steps of adding and removing selected neurons based on indications of local and global activity, using a biased statistical procedure.

Claim 33

The neural network apparatus of claim 1 wherein each portion of said first subset of portions is capable of distinguishing between one set of a first series of sets of attribute measures, the size of said one set of said first series of sets being proportional by a proportionality constant to the number of neurons in said portion of said first subset of portions, where said proportionality constant is approximately 0.15, and wherein each portion of said second subset of portions being capable of distinguishing between one set of a second series of sets of attribute measures, the size of said one set of said second series of sets being proportional by a proportionality constant to the number of neurons in said portion of said second subset of portions, where said proportionality constant is approximately 0-15.

Claim 34

Computer apparatus for processing input information for a prespecified application to indicate one output category from a set of prespecified output categories for said prespecified application in response to said input information, the input information being supplied to said apparatus as a data array, the computer apparatus comprising: an input stage for accepting said data array and converting said data array to a corresponding internal representation; a data preprocessor connected to said input stage for analyzing said data array based on a plurality of distinct feature attributes to generate a corresponding plurality of attribute measures; a plurality of interconnected processors, said processors being interconnected and exercising a mutual influence each processor of a first subset of said processors being prespecified with a particular correspondence to one of said feature attributes and adapted to accept from said data preprocessor one of said attribute measures for said data array, said interconnected processors processing said plurality of attribute measures to generate a result representative of category attributes, each processor of a second subset of said processors being prespecified with a particular correspondence to one of said category attributes; a data postprocessor to said plurality of processors for obtaining category attribute indicators for said category attributes by correlating said result with predefined category attribute measures; and an output stage for accepting said category attribute indicators and combining said category attribute indicators in a prespecified manner to generate said output category for said input information.

Claim 35

Computer apparatus for processing input information for a prespecified application to indicate one output category from a set of prespecified output categories for said prespecified application in response to said input information, the input information being supplied to said apparatus as a data array, the computer apparatus comprising: an input stage for accepting said data array and converting said data array to a corresponding internal representation; a data preprocessor connected to said input stage for analyzing said data array based on a plurality of distinct feature attributes to generate a corresponding plurality of attribute measures; a plurality of interconnected processors, said processors being interconnected and exercising a mutual influence each processor of a first subset of said processors being prespecified with a particular correspondence to one of said feature attributes and adapted to accept from said data preprocessor one of said attribute measures for said data array, said interconnected processors processing said plurality of attribute measures to generate a result representative of category attributes, each processor of a second subset of said processors being prespecified with a particular correspondence to one of said category attributes; a data postprocessor to said plurality of processors for obtaining category attribute indicators for said category attributes by correlating said result with predefined category attribute measures; and an output stage for accepting said category attribute indicators and combining said category attribute indicators in a prespecified manner to generate said output category for said input information.

Claim 36

Neural network apparatus for pattern recognition by categorization of an input pattern by indicating one output category from a set of prespecified output categories in response to said input pattern, said input pattern being supplied to the apparatus as a data stream corresponding to an input pattern field, said neural network apparatus comprising: an input stage for accepting said input pattern and converting said data stream to a corresponding internal representation; a data preprocessor connected to said input stage for analyzing said data stream based on a plurality of distinct pattern attributes to generate a corresponding plurality of attribute measures; a neural network comprising a plurality of interconnected neurons, said neural network divided into portions, each of said portions being predefined to include a number of said interconnected neurons and said portions being interconnected and exercising a mutual influence, each portion of a first subset of said portions being prespecified with a particular correspondence to one of said pattern attributes and adapted to accept from said data preprocessor one of said attribute measures for said input pattern, said neural network processing said attribute measures to reach a neural state representative of pattern category attributes, each portion of a second subset of said portions being prespecified with a particular correspondence to said category attributes; a data postprocessor connected to said neural network for obtaining category attribute indicators for said category attributes by correlating said neural state with predefined pattern category attribute measures; and an output stage for accepting said pattern category attribute indicators from said postprocessor and combining said pattern category attribute indicators in a prespecified manner to generate said output category for said input pattern.

Claim 37

Neural network apparatus for pattern recognition by categorization of an input pattern by indicating one output category from a set of prespecified output categories in response to said input pattern, said input pattern being supplied to the apparatus as a data stream corresponding to an input pattern field, said neural network apparatus comprising: an input stage for accepting said input pattern and converting said data stream to a corresponding internal representation; a data preprocessor connected to said input stage for analyzing said data stream based on a plurality of distinct pattern attributes to generate a corresponding plurality of attribute measures; a neural network comprising a plurality of interconnected neurons, said neural network divided into portions, each of said portions being predefined to include a number of said interconnected neurons and said portions being interconnected and exercising a mutual influence, each portion of a first subset of said portions being prespecified with a particular correspondence to one of said pattern attributes and adapted to accept from said data preprocessor one of said attribute measures for said input pattern, said neural network processing said attribute measures to reach a neural state representative of pattern category attributes, each portion of a second subset of said portions being prespecified with a particular correspondence to said category attributes; a data postprocessor connected to said neural network for obtaining category attribute indicators for said category attributes by correlating said neural state with predefined pattern category attribute measures; and an output stage for accepting said pattern category attribute indicators from said postprocessor and combining said pattern category attribute indicators in a prespecified manner to generate said output category for said input pattern.