US6532454B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Stable adaptive control using critic designs
Assignee: IPU POWER MANAGEMENT LLC
•
Filing: 1999-09-23
•
Granted: 2003-03-11
•
Expires: 2019-09-23
Abstract
Abstract Classical adaptive control proves total-system stability for control of linear plants, but only for plants meeting very restrictive assumptions. Approximate Dynamic Programming (ADP) has the potential, in principle, to ensure stability without such tight restrictions. It also offers nonlinear and neural extensions for optimal control, with empirically supported links to what is seen in the brain. However, the relevant ADP methods in use today—TD, HDP, DHP, GDHP—and the Galerkin-based versions of these all have serious limitations when used here as parallel distributed real-time learning systems. Either they do not possess quadratic unconditional stability or they lead to incorrect results in the stochastic case. (ADAC or Q-learning designs do not help.) The present invention describes new ADP designs which overcome these limitations. It also addresses the Generalized Moving Target problem, a common family of static optimization problems, and describes a way to stabilize large-scale economic equilibrium models, such as the old long-term energy model of DOE.
Inventors
- 👤 Paul J. Werbos
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-05 08:43
CPC Classifications
G; G05; G05B; G05B13/00; G05B13/02; G05B13/0265; G05B13/027; G05B13/04; G05B13/048; G06; G06N; G06N20/00
Claims Analysis
Total Claims: 7
Claim 1
Independent Claim
113 words
In a computer program product including a computer storage medium, the improvement comprising a computer program code mechanism embedded in the computer storage medium for causing a microprocessor to implement a heuristic dynamic programming controller controlling an external system, the computer program code mechanism comprising: a first computer code device configured to train a deterministic forecasting model f′(R(t)) to predict the expectation value of R (t+1); a second computer code device configured to calculate an error function, e, based on sampled value of Ĵ(t+1); and a third computer code device configured to calculate ∇ t e′ by backpropagating through: e′=(Ĵ( R (t,W)−(Ĵ( f ′( R (t)),W)+U)) such that <∂ t W>=−<e>∇ t e′.
Claim 2
Dependent Claim
113 words
In a computer program product including a computer storage medium, the improvement comprising a computer program code mechanism embedded in the computer storage medium for causing a microprocessor to implement a heuristic dynamic programming controller controlling an external system, the computer program code mechanism comprising: a first computer code device configured to train a deterministic forecasting model f′(R(t)) to predict the expectation value of R (t+1); a second computer code device configured to calculate an error function, e, based on sampled value of Ĵ(t+1); and a third computer code device configured to calculate ∇ t e′ by backpropagating through: e′=(Ĵ( R (t,W)−(Ĵ( f ′( R (t)),W)+U)) such that <∂ t W>=−<e>∇ t e′.
Claim 3
Dependent Claim
31 words
In the computer program product as claimed in claim 1 , the improvement further comprising: a fourth code device configured to calculate: ΔW=−2αe(∇ t e′)=2αe(∇Ĵ( R (t))−(I/(1+r))∇Ĵ( f ′( R (t))).
Claim 4
Dependent Claim
187 words
In a computer program product including a computer storage medium, the improvement comprising a computer program code mechanism embedded in the computer storage medium for causing a microprocessor to implement a heuristic dynamic programming controller controlling an external system, the computer program code mechanism comprising: a first computer code device configured to train a deterministic forecasting model f′(R(t)) to predict the expectation value of R (t+1); a second computer code device configured to obtain a value R(t+1) at each time t+1; a third computer code device configured to backpropagated an error, e, through Ĵ( R (t+1)) back to weights of a critic network; a fourth computer code device configured to obtain a first set of derivatives, g1, representing a gradient of e with respect to the weights of the critic network; a fifth computer code device configured to backpropagate e′ through Ĵ( f ′( R (t)), back to the weights of the Critic network to form a second set of derivatives g2; and a sixth computer code device configured to adapt weights of f′ so as to minimize: E g =|g 1 −g 2 | 2 .
Claim 5
Dependent Claim
187 words
In a computer program product including a computer storage medium, the improvement comprising a computer program code mechanism embedded in the computer storage medium for causing a microprocessor to implement a heuristic dynamic programming controller controlling an external system, the computer program code mechanism comprising: a first computer code device configured to train a deterministic forecasting model f′(R(t)) to predict the expectation value of R (t+1); a second computer code device configured to obtain a value R(t+1) at each time t+1; a third computer code device configured to backpropagated an error, e, through Ĵ( R (t+1)) back to weights of a critic network; a fourth computer code device configured to obtain a first set of derivatives, g1, representing a gradient of e with respect to the weights of the critic network; a fifth computer code device configured to backpropagate e′ through Ĵ( f ′( R (t)), back to the weights of the Critic network to form a second set of derivatives g2; and a sixth computer code device configured to adapt weights of f′ so as to minimize: E g =|g 1 −g 2 | 2 .
Claim 6
Dependent Claim
45 words
In the computer program product as claimed in claim 3 , wherein the second computer code device further comprises a seventh computer code device configured to obtain R(t+1) by simulating a random vector w inserted into a stochastic model f ( R (t), w ).
Claim 7
Dependent Claim
37 words
In the computer program product as claimed in claim 3 , wherein the second computer code device further comprises a seventh computer code device configured to obtain R(t+1) by actually observing R (t+1) and estimating w .
References Cited (Backward Citations)
No citation records indexed.
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-03-10 | 2006-09-09 | 2007-03-11 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-03-10 | 2010-09-09 | 2011-03-11 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-03-10 | 2014-09-09 | 2015-03-11 | $7700.00 | Window Closed / Lapsed |