Patent US6516309B1 successfully refreshed from Google Patents.
US6516309B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for evolving a neural network
Assignee: Indiana University Research and Technology Corp
•
Filing: 1999-07-14
•
Granted: 2003-02-04
•
Expires: 2019-07-14
Abstract
Abstract A method of evolving a neural network that includes a plurality of processing elements interconnected by a plurality of weighted connections includes the step of obtaining a definition for the neural network by evolving a plurality of weights for the plurality of weighted connections, and evolving a plurality of activation function parameters associated with the plurality of processing elements. Another step of the method includes determining whether the definition for the neural network may be simplified based upon at least one activation function parameter of the plurality of activation function parameters. Yet another step of the method includes updating the definition for the neural network in response to determining that the definition for the neural network may be simplified. The method utilizes particle swarm optimization techniques to evolve the plurality of weights and the plurality of activation parameters. Moreover, the method simplifies activation functions of processing elements in response to corresponding activation parameters meeting certain criteria, and removes processing elements from the definition of the neural network in response to corresponding activation parameters satisfying certain criteria. Various apparatus are also disclosed for implementing network evolution and simplification.
Inventors
- 👤 Russell C. Eberhart
- 👤 Yuhui Shi
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 20:40
CPC Classifications
G; G06; G06N; G06N3/00; G06N3/02; G06N3/08; G06N3/086
Claims Analysis
Total Claims: 25
Claim 1
Independent Claim
135 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, b) determining, based upon a first activation function parameter of said plurality of activation function parameters, whether a first processing element of said plurality of processing elements may be removed from said neural network to simplify the definition for said neural network, and c) updating said definition for said neural network by removing said first processing element from said definition of said neural network in response to determining that said first processing element may be removed.
Claim 2
Independent Claim
135 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, b) determining, based upon a first activation function parameter of said plurality of activation function parameters, whether a first processing element of said plurality of processing elements may be removed from said neural network to simplify the definition for said neural network, and c) updating said definition for said neural network by removing said first processing element from said definition of said neural network in response to determining that said first processing element may be removed.
Claim 3
Independent Claim
137 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, b) determining whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and c) updating said definition for said neural network by removing a first processing element implementing a sigmoid activation function from said definition of said neural network in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 4
Independent Claim
137 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, b) determining whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and c) updating said definition for said neural network by removing a first processing element implementing a sigmoid activation function from said definition of said neural network in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 5
Dependent Claim
176 words
The method of claim 1 , wherein: step b) further comprises the step of determining that a first processing element of said plurality of processing elements substantially generates a substantially constant output signal regardless of received input signals based upon a first slope factor of a said plurality of activation functions parameters that is associated with a first activation function implemented by said first processing element; and step c) further comprises the steps of: c1) removing said first processing element from said definition of said neural network in response to determining that said first processing element generates said substantially constant output signal, and c2) updating biasing weighted connections of said plurality of weighted connections associated with a biasing processing element of said plurality of processing elements based upon first weighted connections of said plurality of weighted connections associated with said first processing element in order to substantially reproduce an effect said first processing element had on said plurality of processing elements prior to said first processing element being removed from said definition of said neural network.
Claim 6
Independent Claim
305 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, said obtaining the definition for said neural network further comprising a1) initializing a swarm of particles in which each particle has a position in a hyperspace that represents a separate definition for said neural network, and a velocity vector that represents motion of said particle through said hyperspace, a2) determining for each particle of said swarm, a fitness value for said respective definition of said neural network, a3) determining based upon said fitness values whether termination criteria have been satisfied, a4) updating for said each particle of said swarm, a personal best value and a personal best position based upon said respective fitness value for said each particle, a5) updating for said each particle of said swarm, a local best value and a local best position based upon fitness values associated with a respective group of said particles, a6) updating for said each particle of said swarm, said position and said velocity vector for said particle based upon said personal best position for said particle, said local best position for said particle, and said velocity vector for said particle, and a7) repeating steps a2), a3), a4), a5), and a6) until said termination criteria have been satisfied; b) determining whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters; and c) updating said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 7
Independent Claim
305 words
A method of evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, comprising the steps of: a) obtaining a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements, said obtaining the definition for said neural network further comprising a1) initializing a swarm of particles in which each particle has a position in a hyperspace that represents a separate definition for said neural network, and a velocity vector that represents motion of said particle through said hyperspace, a2) determining for each particle of said swarm, a fitness value for said respective definition of said neural network, a3) determining based upon said fitness values whether termination criteria have been satisfied, a4) updating for said each particle of said swarm, a personal best value and a personal best position based upon said respective fitness value for said each particle, a5) updating for said each particle of said swarm, a local best value and a local best position based upon fitness values associated with a respective group of said particles, a6) updating for said each particle of said swarm, said position and said velocity vector for said particle based upon said personal best position for said particle, said local best position for said particle, and said velocity vector for said particle, and a7) repeating steps a2), a3), a4), a5), and a6) until said termination criteria have been satisfied; b) determining whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters; and c) updating said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 8
Dependent Claim
47 words
The method of claim 4 , wherein step a6) comprises the step of: updating said each particle of said swarm such that said velocity vector has more effect on early updates of said each particle than said velocity vector has on later updates of said each particle.
Claim 9
Dependent Claim
124 words
A computer readable medium for evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, said computer readable medium comprising code which when executed by a network evolution system causes said network evolution system to: obtain a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements; determine whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters; and update said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 10
Dependent Claim
124 words
A computer readable medium for evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, said computer readable medium comprising code which when executed by a network evolution system causes said network evolution system to: obtain a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements; determine whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters; and update said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 11
Dependent Claim
114 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to: determine whether said definition for said neural network may be simplified by determining, based upon a first activation function parameter of said plurality of activation parameters, whether a first activation function of a first processing element of said plurality of processing elements may be implemented with a less complex activation function, and update said definition for said neural network by replacing said first activation function with said less complex activation function in response to determining that said first activation function may be implemented with said less complex activation function.
Claim 12
Dependent Claim
99 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to: determine whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and update said definition for said neural network by replacing a sigmoid activation function of a first processing element with a step activation function in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 13
Dependent Claim
102 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to: determine whether said definition for said neural network may be simplified by determining, based upon a first activation function parameter of said plurality of activation parameters, whether a first processing element of said plurality of processing elements may be removed from said neural network, and update said definition for said neural network by removing said first processing element from said definition of said neural network in response to determining that said first processing element may be removed.
Claim 14
Dependent Claim
101 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to: determine whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and update said definition for said neural network by removing a first processing element implementing a sigmoid activation function from said definition of said neural network in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 15
Dependent Claim
200 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to: determine whether said definition for said neural network may be simplified by determining that a first processing element of said plurality of processing elements substantially generates a substantially constant output signal regardless of received input signals based upon a first slope factor of a said plurality of activation functions parameters that is associated with a first activation function implemented by said first processing element; and update said definition for said neural network by (i) removing said first processing element from said definition of said neural network in response to determining that said first processing element generates said substantially constant output signal, and (ii) updating biasing weighted connections of said plurality of weighted connections associated with a biasing processing element of said plurality of processing elements based upon first weighted connections of said plurality of weighted connections associated with said first processing element in order to substantially reproduce an effect said first processing element had on said plurality of processing elements prior to said first processing element being removed from said definition of said neural network.
Claim 16
Dependent Claim
218 words
The computer readable medium of claim 6 , wherein said code when executed by said network evolution system further causes said network evolution system to obtain a definition for said neural network by: a) initializing a swarm of particles in which each particle has a position in a hyperspace that represents a separate definition for said neural network, and a velocity vector that represents motion of said particle through said hyperspace; b) determining for each particle of said swarm, a fitness value for said respective definition of said neural network; c) determining based upon said fitness values whether termination criteria have been satisfied; d) updating for said each particle of said swarm, a personal best value and a personal best position based upon said respective fitness value for said each particle; e) updating for said each particle of said swarm, a local best value and a local best position based upon fitness values associated with a respective group of said particles; f) updating for said each particle of said swarm, said position and said velocity vector for said particle based upon said personal best position for said particle, said local best position for said particle, and said velocity vector for said particle; and g) repeating b), c), d), e), and f) until said termination criteria have been satisfied.
Claim 17
Dependent Claim
74 words
The computer readable medium of claim 12 , wherein said code when executed by said network evolution system further causes said network evolution system to update said position and said velocity vector for said each particle of said swarm by: updating said each particle of said swarm such that said velocity vector has more effect on early updates of said each particle than said velocity vector has on later updates of said each particle.
Claim 18
Dependent Claim
122 words
A network evolution system for evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, said network evolution system comprising: a network evolver operable to obtain a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements; and a network simplifier operable to (i) determine whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters, and (ii) update said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 19
Dependent Claim
122 words
A network evolution system for evolving a neural network comprising a plurality of processing elements interconnected by a plurality of weighted connections, said network evolution system comprising: a network evolver operable to obtain a definition for said neural network by evolving a plurality of weights for said plurality of weighted connections, and evolving a plurality of activation function parameters associated with said plurality of processing elements; and a network simplifier operable to (i) determine whether said definition for said neural network may be simplified based upon at least one activation function parameter of said plurality of activation function parameters, and (ii) update said definition for said neural network in response to determining that said definition for said neural network may be simplified.
Claim 20
Dependent Claim
105 words
The network evolution system of claim 14 , wherein said network simplifier is further operable to: determine whether said definition for said neural network may be simplified by determining, based upon a first activation function parameter of said plurality of activation parameters, whether a first activation function of a first processing element of said plurality of processing elements may be implemented with a less complex activation function, and update said definition of said neural network by replacing said first activation function with said less complex activation function in response to determining that said first activation function may be implemented with said less complex activation function.
Claim 21
Dependent Claim
90 words
The network evolution system of claim 14 , wherein said network simplifier is further operable to: determine whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and update said definition of said neural network by replacing a sigmoid activation function of a first processing element with a step activation function in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 22
Dependent Claim
93 words
The network evolution system of claim 14 , wherein said network simplifier is further operable to: determine whether said definition for said neural network may be simplified by determining, based upon a first activation function parameter of said plurality of activation parameters, whether a first processing element of said plurality of processing elements may be removed from said neural network, and update said definition of said neural network by removing said first processing element from said definition of said neural network in response to determining that said first processing element may be removed.
Claim 23
Dependent Claim
92 words
The network evolution system of claim 14 , wherein said network simplifier is further operable to: determine whether said definition for said neural network may be simplified by determining whether a first slope factor of said plurality of activation function parameters has a predetermined relationship to a slope threshold, and update said definition of said neural network by removing a first processing element implementing a sigmoid activation function from said definition of said neural network in response to determining that said first slope factor has said predetermined relationship to said slope threshold.
Claim 24
Dependent Claim
191 words
The network evolution system of claim 14 , wherein said network simplifier is further operable to: determine whether said definition for said neural network may be simplified by determining that a first processing element of said plurality of processing elements substantially generates a substantially constant output signal regardless of received input signals based upon a first slope factor of a said plurality of activation functions parameters that is associated with a first activation function implemented by said first processing element, and update said definition of said neural network by (i) removing said first processing element from said definition of said neural network in response to determining that said first processing element generates said substantially constant output signal, and (ii) updating biasing weighted connections of said plurality of weighted connections associated with a biasing processing element of said plurality of processing elements based upon first weighted connections of said plurality of weighted connections associated with said first processing element in order to substantially reproduce an effect said first processing element had on said plurality of processing elements prior to said first processing element being removed from said definition of said neural network.
Claim 25
Dependent Claim
209 words
The network evolution system of claim 14 , wherein said network evolver is further operable to obtain said definition of said neural network by: a) initializing a swarm of particles in which each particle has a position in a hyperspace that represents a separate definition for said neural network, and a velocity vector that represents motion of said particle through said hyperspace; b) determining for each particle of said swarm, a fitness value for said respective definition of said neural network; c) determining based upon said fitness values whether termination criteria have been satisfied; d) updating for said each particle of said swarm, a personal best value and a personal best position based upon said respective fitness value for said each particle; e) updating for said each particle of said swarm, a local best value and a local best position based upon fitness values associated with a respective group of said particles; f) updating for said each particle of said swarm, said position and said velocity vector for said particle based upon said personal best position for said particle, said local best position for said particle, and said velocity vector for said particle; and g) repeating b), c), d), e), and f) until said termination criteria have been satisfied.
References Cited (Backward Citations)
US5040214A
Cited
US5142590A
Cited
US5014219A
Cited
US5311601A
Cited
US5774357A
Cited
US5875108A
Cited
US5920477A
Cited
US5835901A
Cited
US5724567A
Cited
US5742739A
Cited
US5748783A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-02-03 | 2006-08-05 | 2007-02-04 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-02-03 | 2010-08-05 | 2011-02-04 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-02-03 | 2014-08-05 | 2015-02-04 | $7700.00 | Window Closed / Lapsed |