US5629988A
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for electronic image stabilization
Assignee: Sarnoff Corp
•
Filing: 1994-10-28
•
Granted: 1997-05-13
•
Expires: 2014-10-28
Abstract
Abstract The invention is a system and method for electronic stabilization of a image produced by an electronic imaging device. The input may be any sequence of image frames from an image source, such as a video camera, an IR or X-ray imager, radar, or from a storage medium such as computer disk memory, video tape or a computer graphics generator. The invention can also be used with images from multiple sources when these must be stabilized with respect to one another. The invention uses a feedback loop and second image warp stage to achieve precise image alignment as part of the displacement estimation process. The output of the stabilization system is a modified image sequence in which unwanted components of image motion have been reduced or removed, or in which new motion of a desired form has been introduced. The output may also include information derived from the input sequence, such as estimated frame-to-frame displacement vectors or motion flow fields, and derived image sequences with reduced noise or highlighted change.
Inventors
- 👤 Peter J. Burt
- 👤 Keith J. Hanna
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-28 08:44
CPC Classifications
H; H04; H04N; H04N23/00; H04N23/60; H04N23/68; G; G06; G06T; G06T7/00; G06T7/20; G06T7/254; H04N23/681; H04N23/6811; H04N23/682; H04N23/683
Claims Analysis
Total Claims: 17
Claim 1
Independent Claim
157 words
In an electronic image stabilization method responsive to a sequence of successive pairs of image frames in digital form, wherein said method includes the steps of (a) employing an iteratively-operated feedback loop comprising a first image warper, a displacement estimator, an adder and a memory to reduce a most erroneous initial estimate of displacement between images depicted by each pair of image frames of said sequence to a least erroneous final estimate of displacement, and (b) storing said final estimate of displacement in said memory; the improvement comprising the further steps of: (c) converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model that includes at least one of zoom, parallax translation, rotation, dilation. and lens distortion; and (d) employing a second image warper to warp a certain image frame of each of said successive pairs of image frames in accordance with said output transformation parameters.
Claim 2
Dependent Claim
157 words
In an electronic image stabilization method responsive to a sequence of successive pairs of image frames in digital form, wherein said method includes the steps of (a) employing an iteratively-operated feedback loop comprising a first image warper, a displacement estimator, an adder and a memory to reduce a most erroneous initial estimate of displacement between images depicted by each pair of image frames of said sequence to a least erroneous final estimate of displacement, and (b) storing said final estimate of displacement in said memory; the improvement comprising the further steps of: (c) converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model that includes at least one of zoom, parallax translation, rotation, dilation. and lens distortion; and (d) employing a second image warper to warp a certain image frame of each of said successive pairs of image frames in accordance with said output transformation parameters.
Claim 3
Dependent Claim
78 words
The electronic image stabilization method defined in claim 1, wherein: said certain image frame of each of said successive pairs of image frames corresponds to the then-current image frame of that pair of image frames in said sequence; and the image frame other than said certain image frame of each of said successive pairs of image frames corresponds to the image frame that immediately precedes the then-current image frame of that pair of image frames in said sequence.
Claim 4
Dependent Claim
78 words
The electronic image stabilization method defined in claim 1, wherein: said certain image frame of each of said successive pairs of image frames corresponds to the image frame that immediately precedes the then-current image frame of that pair of image frames in said sequence; and the image frame other than said certain image frame of each of said successive pairs of image frames corresponds to the then-current image frame of that pair of image frames in said sequence.
Claim 5
Dependent Claim
110 words
The electronic image stabilization method defined in claim 1, wherein each of said successive pairs of image frames in said sequence is available at a plurality of resolutions that extend from a given lowest resolution to a given highest resolution; and wherein step (a) comprises the step of: (e) starting with said given lowest resolution image frames, employing said iteratively-operated feedback loop, in turn, on each successively higher resolution image frames to reduce a most erroneous initial estimate of displacement between images depicted by each given lowest resolution pair of image frames to a least erroneous final estimate of displacement depicted by each given highest resolution pair of image frames.
Claim 6
Dependent Claim
56 words
The electronic image stabilization method defined in claim 1, wherein step (a) comprises the step of: (e) employing the final estimate of displacement between images depicted by a pair of image frames which precedes a current pair of image frames in said sequence as the initial estimate of displacement of said current pair of image frames.
Claim 7
Dependent Claim
50 words
The electronic image stabilization method defined in claim 1, wherein step (c) comprises the step of: (e) converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model that derives output transformation parameters which form a piece wise linear path.
Claim 8
Dependent Claim
49 words
The electronic image stabilization method defined in claim 1, wherein step (c) comprises the step of: (e) converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model in which low pass filtering is applied to smooth image differences.
Claim 9
Dependent Claim
153 words
In an electronic image stabilization apparatus responsive to a sequence of successive pairs of image frames in digital form; wherein said apparatus includes (a) an iteratively-operated feedback loop comprising a first image warper, a displacement estimator, an adder and a memory for reducing a most erroneous initial estimate of displacement between images depicted by each pair of image frames of said sequence to a least erroneous final estimate of displacement, and (b) means for storing said final estimate of displacement in said memory; the improvement comprising: means for converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model that includes at least one of zoom, parallax translation, rotation, dilation and lens distortion; and means for employing a second image warper to warp a certain image frame of each of said successive pairs of image frames in accordance with said output transformation parameters.
Claim 10
Dependent Claim
153 words
In an electronic image stabilization apparatus responsive to a sequence of successive pairs of image frames in digital form; wherein said apparatus includes (a) an iteratively-operated feedback loop comprising a first image warper, a displacement estimator, an adder and a memory for reducing a most erroneous initial estimate of displacement between images depicted by each pair of image frames of said sequence to a least erroneous final estimate of displacement, and (b) means for storing said final estimate of displacement in said memory; the improvement comprising: means for converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model that includes at least one of zoom, parallax translation, rotation, dilation and lens distortion; and means for employing a second image warper to warp a certain image frame of each of said successive pairs of image frames in accordance with said output transformation parameters.
Claim 11
Dependent Claim
57 words
The electronic image stabilization apparatus defined in claim 8, further comprising: a first image frame store for storing said certain image frame of each of said successive pairs of image frames; and a second image frame store for storing the image frame other than said certain image frame of each of said successive pairs of image frames.
Claim 12
Dependent Claim
92 words
The electronic image stabilization apparatus defined in claim 9, wherein: said certain image frame of each of said successive pairs of image frames stored in said first image frame store corresponds to the then-current image frame of that pair of image frames in said sequence; and said image frame other than said certain image frame of each of said successive pairs of image frames stored in said first image frame store corresponds to the image frame that immediately precedes the then-current image frame of that pair of image frames in said sequence.
Claim 13
Dependent Claim
92 words
The electronic image stabilization apparatus defined in claim 9, wherein: said certain image frame of each of said successive pairs of image frames stored in said first image frame store corresponds to the image frame that immediately precedes the then-current image frame of that pair of image frames in said sequence; and said image frame other than said certain image frame of each of said successive pairs of image frames stored in said first image frame store corresponds to the then-current image frame of that pair of image frames in said sequence.
Claim 14
Dependent Claim
113 words
The electronic image stabilization apparatus defined in claim 8, further comprising: pyramid means for providing each of said successive pairs of image frames in said sequence at a plurality of resolutions that extend from a given lowest resolution to a given highest resolution; and wherein: said iteratively-operated feedback loop includes means which start by operating on said given lowest resolution image frames and then, in turn, operate on each successively higher resolution image frames to reduce a most erroneous initial estimate of displacement between images depicted by each given lowest resolution pair of image frames to a least erroneous final estimate of displacement depicted by each given highest resolution pair of image frames.
Claim 15
Dependent Claim
61 words
The electronic image stabilization apparatus defined in claim 8, wherein said apparatus further comprises: (e) means for providing said iteratively-operated feedback loop with the final estimate of displacement between images depicted by a pair of image frames which precedes a current pair of image frames in said sequence as the initial estimate of displacement of said current pair of image frames.
Claim 16
Dependent Claim
48 words
The electronic image stabilization apparatus defined in claim 9, wherein: said means for converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model includes means that derives output transformation parameters which form a piece wise linear path.
Claim 17
Dependent Claim
50 words
The electronic image stabilization apparatus defined in claim 8, wherein: said means for converting said final estimate of displacement stored in said memory into output transformation parameters in accordance with a given image-change model includes low-pass filter means that derives output transformation parameters which are effective in smoothing image differences.
References Cited (Backward Citations)
US4835532A
Cited
US4692806A
Cited
US4703514A
Cited
WO1990000334A1
Cited
US5063603A
Cited
US5067014A
Cited
WO1993007585A1
Cited
US5259040A
Cited
US5251271A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2000-05-12 | 2000-11-11 | 2001-05-13 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2004-05-12 | 2004-11-11 | 2005-05-13 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2008-05-12 | 2008-11-11 | 2009-05-13 | $7700.00 | Window Closed / Lapsed |