US5319587A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Computing element for neural networks
Assignee: Boeing Co
•
Filing: 1989-08-07
•
Granted: 1994-06-07
•
Expires: 2009-08-07
Abstract
Abstract A computing element for use in an array in a neural network. Each computing element has K (K>1) input signal terminals, K input backpropagated signal terminals, K output backpropagated signal terminals and at least one output terminal. The input terminals of the computing element located in row i, column j of the array of computing elements receive a sequence of concurrent input signals on K parallel input lines representing a parallel input signal S ij having vector elements (s ij1 , s ij2 , s ij3 , . . . , s ijk ) T . The K input backpropagated signal terminals are coupled to receive an m-dimensional (m<K) backpropagated signal vector characterized to provide a measure of the performance error of the computing element. The computing element comprises a weighting function means responsive to the concurrent input signal S ij for computing a K-dimensional weighting coefficient and a scalar activation signal u ij by computing a K-dimensional weighting-coefficient vector, W ij =(w ij1 , w ij2 , . . . , w ijk ) T , and by forming the vector inner product of the input signal S ij vector elements and the weighting-coefficient vector, W ij . Feedback signals x ijk =w ijk *p ij are provided from the output backpropagated signal terminals where p ij provides a performance error of all columns of computing elements subsequent to this computing element weighted by the gain of the computing element. A nonlinear processor maps successive values of u ij through a nonlinear mapping function, M ij to provide a single-valued output signal y ij . The nonlinear processor responds to an m-dimensional backpropagated signal vector, X ij+1 =(x 1 ,j+1,p ; x 2 ,j+1,p ; x 3 ,j+1,p ; . . . x m ,j+1,p) T characterized to provide a measure of the performance error of all computing elements subsequent to the computing element. Vector X i ,j+1 comprises the pth member of all backpropagated error vectors from the computing elements of column j+1.
Inventors
- 👤 Stanley A. White
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-29 10:30
CPC Classifications
G; G06; G06N; G06N3/00; G06N3/02; G06N3/06; G06N3/063; G06F; G06F2218/00; G06F2218/02
Claims Analysis
Total Claims: 27
Claim 1
Independent Claim
432 words
A computing element comprising: a weighting function means responsive to a sequence of successive sets of corresponding sample input signals, each successive signal having vector elements s ij1 , s ij2 , . . . s ijk for providing a sequence of activation signals, u ij , at an output, said weighting function means comprising at least two weighting function element, each having a first input responsive to a corresponding sample input signal vector element, and a second input responsive to a series of backpropagated signals (p ij ), each weighting function element being configured to: accumulate the successive products of each successive corresponding sample input signal s ijk times a concurrent backpropagated signal, to form a concurrent weighting coefficient signal, and to multiply successive corresponding sample input signal vector elements by the concurrent weighting coefficient signal to provide a weighted input signal; and summation means for summing the concurrent weighted input signals from each weighting function element to provide a series of activation signals u ij ; an adaptive nonlinear processor having an input responsive to the series of activation signals and preselected network signals, and an output, comprising: a first processor coupled to receive said activation signals and update the contents of a first RAM which is configured to provide a sequence of first signal samples and a sequence of second signal samples in response to said activation and network signals; wherein, each said first signal sample has a value representing a point on the amplitude distribution function of said corresponding activation signal sample; each said second signal sample has a value representing the amplitude density function of said corresponding network signal sample, a second processor coupled to receive each said first signal sample and to provide an output signal representing a point on the inverse of the desired output amplitude-distribution function of the corresponding first signal sample; and, control means for receiving said each first signal sample and generating a corresponding control signal sample, each control signal sample having a value representing a point on the derivative of the inverse of the desired output amplitude-distribution function of the output signal sequence; summation means having an plurality of inputs coupled to receive a plurality of respective backpropagated signals for providing a summed backpropagated signal; and combination means having first input responsive to said control signal, a second input responsive to said second signal sample, a third input responsive to said backpropagated signal, for multiplying said first control signal, said second signal sample and said backpropagated signal to provide said internal backpropagated signal, p ij , to said weighting function means.
Claim 2
Dependent Claim
432 words
A computing element comprising: a weighting function means responsive to a sequence of successive sets of corresponding sample input signals, each successive signal having vector elements s ij1 , s ij2 , . . . s ijk for providing a sequence of activation signals, u ij , at an output, said weighting function means comprising at least two weighting function element, each having a first input responsive to a corresponding sample input signal vector element, and a second input responsive to a series of backpropagated signals (p ij ), each weighting function element being configured to: accumulate the successive products of each successive corresponding sample input signal s ijk times a concurrent backpropagated signal, to form a concurrent weighting coefficient signal, and to multiply successive corresponding sample input signal vector elements by the concurrent weighting coefficient signal to provide a weighted input signal; and summation means for summing the concurrent weighted input signals from each weighting function element to provide a series of activation signals u ij ; an adaptive nonlinear processor having an input responsive to the series of activation signals and preselected network signals, and an output, comprising: a first processor coupled to receive said activation signals and update the contents of a first RAM which is configured to provide a sequence of first signal samples and a sequence of second signal samples in response to said activation and network signals; wherein, each said first signal sample has a value representing a point on the amplitude distribution function of said corresponding activation signal sample; each said second signal sample has a value representing the amplitude density function of said corresponding network signal sample, a second processor coupled to receive each said first signal sample and to provide an output signal representing a point on the inverse of the desired output amplitude-distribution function of the corresponding first signal sample; and, control means for receiving said each first signal sample and generating a corresponding control signal sample, each control signal sample having a value representing a point on the derivative of the inverse of the desired output amplitude-distribution function of the output signal sequence; summation means having an plurality of inputs coupled to receive a plurality of respective backpropagated signals for providing a summed backpropagated signal; and combination means having first input responsive to said control signal, a second input responsive to said second signal sample, a third input responsive to said backpropagated signal, for multiplying said first control signal, said second signal sample and said backpropagated signal to provide said internal backpropagated signal, p ij , to said weighting function means.
Claim 3
Dependent Claim
59 words
The signal processor of claim 1 wherein said nonlinear processor further comprises a second RAM, the second RAM being addressed by said activation signal for providing a second signal sample corresponding to each activation signal, the value of each second signal sample corresponding to a point on the amplitude density function of the past activation signal u ij .
Claim 4
Dependent Claim
236 words
An adaptive computing element for use in a network of at least two computing elements arrayed in a predetermined number of (i) rows and (j) columns as determined by a preselected processing size for the network, each computing element comprising: a plurality of input-signal terminals connected to receive concurrent input signals, a same number of output backpropagated signal terminals, at least one element output signal terminal for connecting said element to other network nodes, and a plurality of input backpropagated signal terminals connected to receive backpropagated network signals from each network node to which said output terminal is connected which provide a measure of the performance error of said adaptive computing element, each computing element further comprising: bilateral scaling means for receiving and adjusting the gain of said concurrent input signals in response to an internal backpropagation signal and for generating a scalar activation signal therefrom, and for providing a plurality of network feedback signals on said output backpropagated signal terminals by adjusting the gain of said internal backpropagated error signal; and adaptive nonlinear means for receiving and altering an amplitude density of said scalar activation signal in response to a preselected network control signal for providing a single-valued element output signal at each output terminal, and for receiving and scaling said input backpropagated signals by the instantaneous feedforward gain of the adaptive nonlinear means so as to generate said internal backpropagated performance error signal therefrom.
Claim 5
Dependent Claim
236 words
An adaptive computing element for use in a network of at least two computing elements arrayed in a predetermined number of (i) rows and (j) columns as determined by a preselected processing size for the network, each computing element comprising: a plurality of input-signal terminals connected to receive concurrent input signals, a same number of output backpropagated signal terminals, at least one element output signal terminal for connecting said element to other network nodes, and a plurality of input backpropagated signal terminals connected to receive backpropagated network signals from each network node to which said output terminal is connected which provide a measure of the performance error of said adaptive computing element, each computing element further comprising: bilateral scaling means for receiving and adjusting the gain of said concurrent input signals in response to an internal backpropagation signal and for generating a scalar activation signal therefrom, and for providing a plurality of network feedback signals on said output backpropagated signal terminals by adjusting the gain of said internal backpropagated error signal; and adaptive nonlinear means for receiving and altering an amplitude density of said scalar activation signal in response to a preselected network control signal for providing a single-valued element output signal at each output terminal, and for receiving and scaling said input backpropagated signals by the instantaneous feedforward gain of the adaptive nonlinear means so as to generate said internal backpropagated performance error signal therefrom.
Claim 6
Dependent Claim
43 words
The apparatus of claim 3 wherein said bilateral scaling means comprises: weighting means for applying weighting coefficients to said concurrent input signals and said internal backpropagated performance error signal; and summation means for summing weighted input signals to form said scalar activation signal.
Claim 7
Dependent Claim
39 words
The apparatus of claim 4 wherein said adaptive nonlinear means comprises output means connected to receive said activation signal for providing a corresponding element output signal which is an inverse probability value corresponding to a desired output distribution function.
Claim 8
Dependent Claim
12 words
The apparatus of claim 5 wherein said output means comprises a ROM.
Claim 9
Dependent Claim
79 words
The apparatus of claim 5 wherein said output means comprises: memory means connected to said weighting means for receiving said activation signal and for compiling values thereof, and for providing scalar intermediate output signals corresponding to values on a distribution curve compiled from all previous values of said activation signals; and inverse distribution means connected to receive each intermediate output signal for providing a corresponding output which is an inverse probability value corresponding to the desired output distribution function.
Claim 10
Dependent Claim
34 words
The apparatus of claim 7 wherein said inverse distribution means comprises a first ROM containing prestored inverse probability values and connected to receive said intermediate output signals as addresses for recalling said prestored values.
Claim 11
Dependent Claim
27 words
The apparatus of claim 7 wherein said memory means comprises a first RAM connected to receive and store compiled activation signal data as said distribution curve data.
Claim 12
Dependent Claim
63 words
The apparatus of claim 9 wherein said memory means comprises an accumulator connected to receive said activation signal for compiling values thereof and connected to said first RAM for transferring said accumulated values to predetermined address locations, as said distribution curve data, and said first RAM having address selection inputs connected to said weighting means for receiving said activation signals as address values.
Claim 13
Dependent Claim
46 words
The apparatus of claim 10 wherein said nonlinear means comprises; measuring means for determining an amplitude density function for said preselected neural network signal; and adjusting means connected to said RAM for adjusting data stored therein in response to changes in said amplitude distribution function values.
Claim 14
Dependent Claim
46 words
The apparatus of claim 7 further comprising; backpropagation summation means for summing said input backpropagation signals into a scaler backpropagation signal; and combination means for forming a product of said instantaneous feedforward gain and each successive scaler backpropagation signal to provide said corresponding internal backpropagated signal.
Claim 15
Dependent Claim
32 words
The apparatus of claim 12 further comprising second nonlinear means for applying a conditional nonlinearity to said internal backpropagated signals in response to said intermediate output signals and said network control signal.
Claim 16
Dependent Claim
152 words
The apparatus of claim 13 wherein said second nonlinear means comprises: backpropagation summation means for summing said input backpropagation signals into a scaler backpropagation signal; derivative inverse means connected to said memory means for providing a corresponding scalar signal representing a point on the derivative of the inverse of the probability distribution of the desired output signal in response to each intermediate output signal; probability means connected to receive said activation signals for generating and compiling a probability density function for said activation signals; selecting means connected to receive said activation signals for selecting a scalar probability density signal for each successive activation signal from said compilation of all past values for said activation signals; and multiplication means for forming a product of said inverse of the probability distribution, and derivative of the inverse of the probability distribution scalar signals and said summed backpropagation signals to provide said corresponding internal backpropagated signal.
Claim 17
Dependent Claim
65 words
The apparatus of claim 14 wherein: said derivative inverse means comprises a second ROM storing preselected values for the derivative of the inverse of the probability distribution of the desired output signal at addresses corresponding to values of said intermediate output signals; and said probability means comprises a second RAM storing preselected probability density function values at addresses corresponding to values of said activation signals.
Claim 18
Dependent Claim
22 words
The apparatus of claim 4 wherein said weighting means further comprises biasing means for applying an offset bias to said activation signals.
Claim 19
Dependent Claim
249 words
The apparatus of claim 4 wherein said nonlinear means comprises: an accumulator connected to receive said activation signals from said weighting means and configured to compile values thereof; a first RAM connected to an output of said accumulator so as to receive and store compiled activation signal data in predetermined address locations, as said distribution curve data, said first RAM also having address selection inputs connected to said weighting means for receiving said activation signals as address values which provides scalar intermediate output signals corresponding to values on said distribution curve; a first ROM containing prestored inverse probability values and connected to receive said first RAM output signals as addresses for recalling said prestored values so as to form element output signals; backpropagation summation means for summing said input backpropagation signals into a scaler backpropagation signal; a second ROM connected to said first RAM containing preselected values for the derivative of the inverse of the probability distribution of the desired output signal at addresses corresponding to values of said intermediate output signals; a second RAM connected to receive said activation signals having preselected probability density function values stored therein at addresses corresponding to values of said activation signals; and a multiplier connected to outputs of each of said summation, second RAM, and second ROM, so as to generate said corresponding internal backpropagated signal as a product of said inverse of the probability distribution, and derivative of the inverse of the probability distribution scalar signals and said summed backpropagation signals.
Claim 20
Independent Claim
176 words
A method of performing adaptive nonlinear computations in a computing element within a neural network of computing elements comprising the steps of: receiving and adjusting the gain of a sequence of concurrent input signals in response to internal backpropagation error signals; generating a same number of output backpropagated signals, by adjusting the gain of said internal backpropagated error signals; generating a scalar activation signal from said scaled input signals; receiving and altering an amplitude density of said scalar activation signal in response to a preselected network control signal so as to provide a single-valued element output signal to other network nodes; receiving a plurality of input backpropagated network signals from each network node to which said output is transferred which provide a measure of the performance error of said adaptive computing element; and scaling said input backpropagated signals by the instantaneous feedforward gain associated with said nonlinear computations so as to form said internal backpropagated performance error signal therefrom, which provides a measure of the performance error of all computing elements subsequent to said computing element.
Claim 21
Independent Claim
176 words
A method of performing adaptive nonlinear computations in a computing element within a neural network of computing elements comprising the steps of: receiving and adjusting the gain of a sequence of concurrent input signals in response to internal backpropagation error signals; generating a same number of output backpropagated signals, by adjusting the gain of said internal backpropagated error signals; generating a scalar activation signal from said scaled input signals; receiving and altering an amplitude density of said scalar activation signal in response to a preselected network control signal so as to provide a single-valued element output signal to other network nodes; receiving a plurality of input backpropagated network signals from each network node to which said output is transferred which provide a measure of the performance error of said adaptive computing element; and scaling said input backpropagated signals by the instantaneous feedforward gain associated with said nonlinear computations so as to form said internal backpropagated performance error signal therefrom, which provides a measure of the performance error of all computing elements subsequent to said computing element.
Claim 22
Dependent Claim
48 words
The method of claim 18 wherein said steps of receiving and adjusting and generating an activation signal, comprise the steps of: applying weighting coefficients to said concurrent input signals and said internal backpropagated performance error signal; and summing said weighted input signals to form said scalar activation signal.
Claim 23
Dependent Claim
70 words
The method of claim 18 wherein said receiving and altering an amplitude density step, comprises the steps of: receiving said activation signals and compiling values thereof; providing scalar intermediate output signals corresponding to values on a distribution curve compiled from all previous values of said activation signals; and providing a corresponding output for each intermediate output signal which is an inverse probability value corresponding to a desired output distribution function.
Claim 24
Dependent Claim
84 words
The method of claim 18 wherein said providing step comprise the steps of: storing compiled activation signal data as said distribution curve data in a first RAM at predetermined address locations; applying said activation signals as address values to said first RAM so as to recall said prestored data as intermediate output signals; storing inverse distribution values in a first ROM; and applying said intermediate output signals as address values to said first ROM so as to recall said prestored values as output signals.
Claim 25
Dependent Claim
24 words
The method of claim 21 further comprising the step of updating said first and second RAM data in response to a preselected network signal.
Claim 26
Dependent Claim
109 words
The method of claim 21 further comprising the steps of: summing said input backpropagation signals into a scaler backpropagation signal; storing preselected values for the derivative of the inverse of the probability distribution of the desired output signal in a second ROM connected to said first RAM at addresses corresponding to values of said intermediate output signals; storing preselected probability density function values in a a second RAM connected to receive said activation signals at addresses corresponding to values of said activation signals; and generating a product of the outputs of each of said summation, second RAM, and second ROM, so as to form said corresponding internal backpropagated signals.
Claim 27
Dependent Claim
44 words
The method of claim 18 further comprising the steps of: summing said input backpropagation signals into a scaler backpropagation signal; and forming a product of said instantaneous feedforward gain and each successive scaler backpropagation signal so as to provide said corresponding internal backpropagated signal.
References Cited (Backward Citations)
US3950733A
Cited
US4315319A
Cited
US4507741A
Cited
SU1262538A1
Cited
US4843583A
Cited
US4920499A
Cited
US4918618A
Cited
US4967369A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1997-06-06 | 1997-12-06 | 1998-06-07 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2001-06-06 | 2001-12-06 | 2002-06-07 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2005-06-06 | 2005-12-06 | 2006-06-07 | $7700.00 | Window Closed / Lapsed |