Patent US6003825A successfully refreshed from Google Patents.
Patent US5337395A successfully refreshed from Google Patents.
US5337395A
Expired / Lapsed
US
⏱️ 0.0 years remaining
SPIN: a sequential pipeline neurocomputer
Assignee: International Business Machines Corp
•
Filing: 1991-04-08
•
Granted: 1994-08-09
•
Expires: 2011-04-08
Abstract
Abstract A neural network architecture consisting of input weight multiplications, product summation, neural state calculations, and complete connectivity among the neuron processing elements. Neural networks are modelled using a sequential pipelined neurocomputer producing high performance with minimum hardware by sequentially processing each neuron in the completely connected network model. An N neuron network is implemented using multipliers, a pipelined adder tree structure, and activation functions. The activation functions are provided by using one activation function module and sequentially passing the N input product summations sequentially through it. One bus provides N×N communications by sequentially providing N neuron values to the multiplier registers. The neuron values are ensured of reaching corresponding multipliers through a tag compare function. The neuron information includes a source tag and a valid signal. Higher performance is provided by connecting a number of the neurocomputers in a parallel.
Inventors
- 👤 Stamatis Vassiliadis
- 👤 Gerald G. Pechanek
- 👤 Jose G. Delgado-Frias
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-08 23:28
CPC Classifications
G; G06; G06N; G06N3/00; G06N3/02; G06N3/04; G06N3/044; G06N3/0442
Claims Analysis
Total Claims: 40
Claim 1
Independent Claim
229 words
A sequential pipelined neural computer apparatus comprising: N Processing Elements (PEs), each including storage for a neuron value and for a PE tag, tag matching means for comparing a tag identifying a new neuron value with the PE tag, means for storing the new neuron value as the neuron value of the PE if the tag of the new neuron value matches the PE tag, and N-1 of the PEs having means for passing the new neuron value to a next one of the PEs if the tag identifying the new neuron value does not match the PE tag; N weight storage units, each associated with a respective one of the PEs, and each storing a plurality of weight values; N multipliers, each associated with a respective one of the weight storage units, and each for producing as an output the product of one of the weight values of said respective one of the weight storage units and the neuron value of said respective one of the PEs associated with said respective one of the weight storage units; a summation tree for adding the outputs of the multipliers to produce a sum; a neuron activation function generator, for receiving the sum and generating therefrom a new neuron value; and means for communicating the generated neuron value from the generator to one of the PEs as the new neuron value.
Claim 2
Dependent Claim
229 words
A sequential pipelined neural computer apparatus comprising: N Processing Elements (PEs), each including storage for a neuron value and for a PE tag, tag matching means for comparing a tag identifying a new neuron value with the PE tag, means for storing the new neuron value as the neuron value of the PE if the tag of the new neuron value matches the PE tag, and N-1 of the PEs having means for passing the new neuron value to a next one of the PEs if the tag identifying the new neuron value does not match the PE tag; N weight storage units, each associated with a respective one of the PEs, and each storing a plurality of weight values; N multipliers, each associated with a respective one of the weight storage units, and each for producing as an output the product of one of the weight values of said respective one of the weight storage units and the neuron value of said respective one of the PEs associated with said respective one of the weight storage units; a summation tree for adding the outputs of the multipliers to produce a sum; a neuron activation function generator, for receiving the sum and generating therefrom a new neuron value; and means for communicating the generated neuron value from the generator to one of the PEs as the new neuron value.
Claim 3
Dependent Claim
16 words
The apparatus according to claim 1 wherein the neuron activation function is a nonlinear sigmoid generator.
Claim 4
Dependent Claim
19 words
The apparatus according to claim 1 wherein the summation tree contains log 2 N 2 to 1 adder stages.
Claim 5
Dependent Claim
31 words
The apparatus according to claim 3 further comprising means for pipelining the summation tree to allow a new summation process to start prior to a previous summation finishing on the tree.
Claim 6
Dependent Claim
38 words
The apparatus according to claim 1 wherein the means for communicating is a bus comprising a packet of neuron information made up of the neuron value, a Source tag, a Valid signal, and a host-only controlled Weight bit.
Claim 7
Dependent Claim
32 words
The apparatus according to claim 5 further comprising a pipeline control mechanism for generating the Valid signals and Source tags to be communicated in synchronism with the neuron value on the bus.
Claim 8
Dependent Claim
49 words
The apparatus according to claim 1 further comprising a host interface control for providing: a separate interface to a host computer, and means for attaching the host computer to the means for communicating for initialization of the apparatus, and for access to the weight values and the neuron values.
Claim 9
Dependent Claim
43 words
The apparatus according to claim 1 wherein the weight storage units have: means for addressing the N weight values, a common N address generator for all weight arrays, wherein said address generator provides a Nth event output to indicate a Nth update cycle.
Claim 10
Dependent Claim
31 words
The apparatus according to claim 1 comprising means for sequentially generating and outputting the new neuron values onto the means for communicating in an order specified by the PEs' stored tags.
Claim 11
Dependent Claim
41 words
The apparatus according to claim 1 further comprising a valid bit placed on the means for communicating with the new neuron value and its identifying tag for one control period to ensure that only valid and stable neuron information is communicated.
Claim 12
Dependent Claim
19 words
The apparatus according to claim 1 further comprising means for inactivating the means for storing the new neuron values.
Claim 13
Dependent Claim
38 words
The apparatus according to claim 1 further comprising means for indicating that a last new neuron value has been communicated to the one of the PEs to indicate when a next set of parallel multiplications can be started.
Claim 14
Dependent Claim
37 words
The apparatus according to claim 1 wherein the means for comparing in the PEs is eliminated and the new neuron values are stored as the neuron value of the PE upon an indication from a counter mechanism.
Claim 15
Dependent Claim
16 words
The apparatus according to claim 1 further comprising an accumulating adder connected to the summation tree.
Claim 16
Dependent Claim
45 words
The apparatus according to claim 14 wherein said accumulating adder contains (m) storage elements, a Partial Sum Array, for storing (m) Partial Summation (PS) values, and where the neuron activation function generator is not engaged until a full input summation dictated by ##EQU35## is met.
Claim 17
Dependent Claim
97 words
The apparatus according to claim 14 wherein there is provided an expanded weight array for larger networks containing (m) neurons, and wherein an apparatus control structure is provided to follow an operation sequence for a given size of weight memory. MEM weights, whereby at each of N apparatus inputs and for a given N, neural networks where m>N can be executed, the size of (m) is given by: m=√(N(MEM)) and wherein there are a possible m 2 weights in a completely connected (m) neuron network and the m 2 weights are divided among the N apparatus inputs.
Claim 18
Dependent Claim
31 words
The apparatus according to claim 16 further comprising means for sending all the neuron values to a HOST which would initialize the PEs with corresponding neuron values and START the network.
Claim 19
Dependent Claim
26 words
The apparatus of claim 1 connected in parallel with E-1 substantially similar apparatuses, and further comprising an interconnection network of E buses, and means for controlling.
Claim 20
Dependent Claim
43 words
The apparatus according to claim 18 wherein the E parallel apparatuses share PEs in a serially connected N PE structure such that the PE's storage for a neuron value is connected in parallel to their corresponding multiplier inputs in the E parallel apparatuses.
Claim 21
Dependent Claim
45 words
The apparatus according to claim 19 wherein there is a synchronization mechanism, provided as input to the serially connected N PEs, that orders neuron value outputs obtained from the E buses and inputs the ordered neuron value to the means for passing the neuron value.
Claim 22
Dependent Claim
21 words
The apparatus according to claim 18 further comprising a bus containing N/E Y-values produced by the parallel apparatuses' neuron activation functions.
Claim 23
Dependent Claim
77 words
The apparatus according to claim 18 further comprising weight storage units containing weight values sequentially ordered such that the neuron outputs are produced in a sequential order for each parallel apparatus identified as PN-K where K is equal to 1, 2, . . . , E, and the weight address "i" sequentially varies from 1, 2, . . . , N/E to produce Source Tags in each parallel apparatus as given by PN-K Tags equal to K+E(i-1).
Claim 24
Dependent Claim
16 words
The apparatus according to claim 1 wherein there are plural unpipelined multipliers and neuron activation functions.
Claim 25
Dependent Claim
44 words
The apparatus according to claim 23 wherein there are A*N multipliers and B neuron activation functions, where A is a dimensionless number equal to a multiplier clock cycle delay and B is a dimensionless number equal to a neuron activation function clock cycle delay.
Claim 26
Dependent Claim
64 words
The apparatus according to claim 24 wherein N pipelined multipliers and one pipelined neuron activation function are used with A stages in each multiplier and B stages in the single neuron activation function with one bus where A is a dimensionless number equal to the multiplier clock cycle delay and B is a dimensionless number equal to the neuron activation function clock cycle delay.
Claim 27
Dependent Claim
23 words
The apparatus according to claim 23 wherein there is a common input to the neuron activation functions with sequential non-overlapping summation value inputs.
Claim 28
Dependent Claim
32 words
The apparatus according to claim 23 wherein the neuron activation functions produce neuron values one clock cycle apart from one another, and wherein a single L bit wide parallel bus is provided.
Claim 29
Dependent Claim
42 words
The apparatus according to claim 23 further comprising bit serial multipliers and summation tree values and wherein the values are accumulated in the summation tree in bit-serial form and transmitted to the neuron activation function generators at a one clock cycle rate.
Claim 30
Dependent Claim
34 words
The apparatus according to claim 23 wherein the summation trees have 2 to 1 adders and bit-serial summation values are accumulated in the tree prior to providing input to the neuron activation function generators.
Claim 31
Dependent Claim
35 words
The apparatus according to claim 29 wherein multiple summation trees are provided with bit accumulators located at their outputs, and wherein accumulated outputs of the summation trees are distributed to B neuron activation function inputs.
Claim 32
Dependent Claim
17 words
The apparatus according to claim 23 wherein a single summation tree is provided for word parallel processing.
Claim 33
Dependent Claim
37 words
The apparatus according to claim 31 wherein there is included A multipliers per input with each multiplier feeding to an A to 1 multiplexor, and the output of the multiplexor provides input to the single summation tree.
Claim 34
Dependent Claim
45 words
The apparatus according to claim 1 wherein there are E substantiality similar parallel apparatuses, E buses, X replicated input multiplier sections contained in E-1 apparatuses for producing P neuron values from the E-1 parallel apparatuses, where E, X, and P are integers greater than one.
Claim 35
Dependent Claim
34 words
The apparatus according to claim 33 wherein Y, an integer greater than one, neuron values are sequentially output from each apparatus, each value having an identifying tag and placed on the means for communicating.
Claim 36
Dependent Claim
87 words
The apparatus according to claim 34 wherein there is provided a queue store, and wherein the values are placed in said queue, and wherein there are means to sequentially multiply the Y neuron values from the queue with their associated weights, and means to sum the products by use of summation trees with iterative adders, whereby a final iterative sum represents the summation of all weighted inputs for a N th neuron and its value is input to a neuron activation function to develop a neuron value.
Claim 37
Dependent Claim
29 words
The apparatus according to claim 35 wherein P neuron values are produced from the E-1 parallel apparatuses and N-P neuron values are produced from the E th parallel apparatus.
Claim 38
Dependent Claim
39 words
The apparatus according to claim 35 wherein the queue store comprises two storage arrays, and wherein one array keeps present cycle neuron values, and the other is updated with new neuron values used in a subsequent network update cycle.
Claim 39
Dependent Claim
22 words
The apparatus according to claim 37 wherein the two storage arrays function as queue buffers which are switched for each update cycle.
Claim 40
Dependent Claim
39 words
The apparatus according to claim 33 wherein, for reduced hardware complexity, the neuron value storage are shared between the parallel apparatuses using N/X to 1 multiplexors of negligible delay to provide the input to E-1 parallel apparatus input multipliers.
References Cited (Backward Citations)
US4994982A
Cited
US4897814A
Cited
US5075889A
Cited
US5091864A
Cited
US5040134A
Cited
US5115492A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1997-08-08 | 1998-02-07 | 1998-08-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2001-08-08 | 2002-02-07 | 2002-08-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2005-08-08 | 2006-02-07 | 2006-08-09 | $7700.00 | Window Closed / Lapsed |