Patent US6529613B1 successfully refreshed from Google Patents.
US6529613B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Motion tracking using image-texture templates
Assignee: Disney Enterprises Inc
•
Filing: 1997-11-26
•
Granted: 2003-03-04
•
Expires: 2017-11-26
Abstract
Abstract Image templates are extracted from video images in real-time and stored in memory. Templates are selected on the basis of their ability to provide useful positional data and compared with regions of subsequent images to find the position giving the best match. From the position data a transform model is calculated. The transform model tracks the background motion in the current image to accurately determine the motion and attitude of the camera recording the current image. The transform model is confirmed by examining pre-defined image templates. Transform model data and camera sensor data are then used to insert images into the live video broadcast at the desired location in the correct perspective. Stored templates are periodically updated to purge those that no longer give valid or significant positional data. New templates extracted from recent images are used to replace the discarded templates.
Inventors
- 👤 Brian Astle
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:08
CPC Classifications
H; H04; H04N; H04N5/00; H04N5/222; H04N5/262; H04N5/272; H04N5/2723; G; G01; G01S; G01S3/00; G01S3/78; G01S3/782; G01S3/785; G01S3/786; G01S3/7864; G06; G06T; G06T7/00; G06T7/20; G06T7/246; G06V; G06V10/00; G06V10/20; G06V10/24; G06V10/245; G01S17/00; G01S17/66; H04N5/14; H04N5/144; H04N5/145
Claims Analysis
Total Claims: 26
Claim 1
Independent Claim
156 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said set of image templates comprises an array of adjacent pixels from a current video image and wherein said image templates comprise substantially less than all of the pixels in said current video image.
Claim 2
Independent Claim
156 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said set of image templates comprises an array of adjacent pixels from a current video image and wherein said image templates comprise substantially less than all of the pixels in said current video image.
Claim 3
Independent Claim
154 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said purging step (d) and said obtaining step (a) are performed in either the odd video fields only or the even video fields only in order to reduce video interlace stability problems.
Claim 4
Independent Claim
154 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said purging step (d) and said obtaining step (a) are performed in either the odd video fields only or the even video fields only in order to reduce video interlace stability problems.
Claim 5
Independent Claim
148 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said obtaining step (a) is terminated after a preset time limit or after a sufficient number of image templates have been obtained, whichever event occurs first.
Claim 6
Independent Claim
148 words
A method for tracking motion in a stream of video images comprising the steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform model being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; and e) obtaining new image templates from said current image to replace the image templates that were purged, wherein said obtaining step (a) is terminated after a preset time limit or after a sufficient number of image templates have been obtained, whichever event occurs first.
Claim 7
Independent Claim
178 words
A method for tracking motion in a stream of video images comprising steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; f) obtaining new image templates from said current image to replace the image templates that were purged; g) determining an error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position; h) evaluating said error surface to determine its minimum value in order to determine the best match between the image templates and the current image; and, i) using said error surface in said transform model calculations.
Claim 8
Independent Claim
178 words
A method for tracking motion in a stream of video images comprising steps of: a) obtaining a set of image templates (block) from a current video image that meet certain template capturing criteria and storing said image templates in memory; b) determining the position of each stored image template with respect to the current image; c) calculating a transform model using the determined template positions with respect to the current image, said transform being a descriptor of the change of position of said set of image templates; d) purging image templates from memory that do not meet certain template retention criteria; f) obtaining new image templates from said current image to replace the image templates that were purged; g) determining an error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position; h) evaluating said error surface to determine its minimum value in order to determine the best match between the image templates and the current image; and, i) using said error surface in said transform model calculations.
Claim 9
Dependent Claim
31 words
The method of claim 1 further comprising the step of: i) confirming the accuracy of said transform model by comparing its correspondence results against a set of previously defined synthetic templates.
Claim 10
Dependent Claim
77 words
The method of claim 5 wherein determining the position of each stored image template with respect to the current image comprises the steps of: j) performing an integer Position search in order to determine the minimum value of said error surface; and k) upon completion of said integer position search, performing a fractional pixel interpolation in order to estimate the fractional part of the motion of a small piece of the previous image to the current image.
Claim 11
Dependent Claim
143 words
The method of claim 6 wherein performing the integer position search comprises the steps of: l) positioning each template at several integer position locations in the image template and calculating an error surface for each location using the specified block matching criteria m) searching a series of template regions having an initial search region centered about a predicted template position derived from an estimation of the motion of a small piece of the previous image to the current image, said searching to determine image template size and shape; n) terminating the search successfully if a minimum is found within the predicted search region; and o) terminating the integer position search unsuccessfully if, after several at tempts, a minimum cannot be found within the predicted search region and storing said information pertaining to the unsuccessful search so that the block may be later purged.
Claim 12
Dependent Claim
14 words
The method of claim 7 wherein said integer position search uses linear prediction techniques.
Claim 13
Dependent Claim
15 words
The method of claim 7 wherein said integer position search uses second-order polynomial prediction techniques.
Claim 14
Dependent Claim
88 words
The method of claim 7 wherein said error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position is calculated according to the following block matching technique: M = 1 - N = 1 - ∑ I T ∑ I 2 ∑ T 2 where M represents the mismatch value, N represents a normalized correlation calculation, I represents the pixel values in the current image, and T represents the pixel values in the image template.
Claim 15
Dependent Claim
93 words
The method of claim 7 wherein said error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position is calculated according to the following block matching technique: M = 1 - B A = 1 - 2 * ∑ I T ∑ I 2 + ∑ T 2 where M represents the mismatch value, BA represents an error surface calculation, I represents the pixel values in the current image, and T represents the pixel values in the image template.
Claim 16
Dependent Claim
77 words
The method of claim 7 wherein said error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position is calculated according to the following block matching technique: M = 1 - L1norm = 1 - ∑ d where M represents the mismatch value, L 1 norm represents an error surface calculation, and d represents the difference in pixel values between the image template and the current image.
Claim 17
Dependent Claim
15 words
The method of claim 12 wherein the fractional pixel interpolation uses a triangular interpolation method.
Claim 18
Dependent Claim
78 words
The method of claim 7 wherein said error surface indicating mismatches between each image template and the current image in a region proximate to the determined template position is calculated according to the following block matching technique: M = 1 - L2norm = 1 - ∑ d 2 where M represents the mismatch value, L 2 norm represents an error surface calculation, and d represents the difference in pixel values between the image template and the current image.
Claim 19
Dependent Claim
15 words
The method of claim 14 wherein the fractional pixel interpolation uses a parabolic interpolation method.
Claim 20
Dependent Claim
16 words
The method of claim 7 wherein the fractional pixel interpolation uses a three-halves power interpolation method.
Claim 21
Dependent Claim
61 words
The method of claim 7 wherein said evaluating said error surface to determine its minimum value in order to determine the best match between the image templates and the current image comprises the steps of: p) expanding the image template yielding sub-pixel position values; and q) performing a further integer position search according to step (j) above at these sub-pixel locations.
Claim 22
Dependent Claim
163 words
The method of claim 7 wherein said evaluating said error surface to determine its minimum value in order to determine the best match between the image templates and the current image comprises the steps of: r) obtaining error surface values proximate to the minimum value determined by the integer position search of step (j); s) interpolating for a value at the horizontal scan line just above where the original integer position search determined a minimum, said interpolating carried out by a one dimensional method; t) interpolating for a value at the horizontal scan line where the original integer position search determined a minimum, said interpolating carried out by a one dimensional method; u) interpolating for a value at the horizontal scan line just below where the original integer position search determined a minimum, said interpolating carried out by a one dimensional method; and v) interpolating the values from steps (s), (t), and (u) to determine a final minimum value for said error surface.
Claim 23
Dependent Claim
48 words
The method of claim 7 wherein said evaluating said error surface to determine its minimum value in order to determine the best match between the image templates and the current image comprises the step of: w) interpolating the position of the minimum using a singular value decomposition method.
Claim 24
Dependent Claim
86 words
The method of claim 7 wherein calculating said transform model comprises the steps of: x) setting horizontal and vertical weights for each block depending on the curvature of the error surface; y) calculating a preliminary transform model using a mean square error minimization method; z) evaluating each block for spatial error to determine how well it agrees with the preliminary transform model; aa) modifying said weights for each block according to the spatial error; and bb) calculating a final transform model using the modified block weights.
Claim 25
Dependent Claim
56 words
The method of claim 7 wherein said template retention criteria require that image templates, in order not to be purged, must not be in a horizontal blanking area, must not be in an active burn-in region, must agree with the current transform model with respect to position, and must have sufficient curvature of the error surface.
Claim 26
Dependent Claim
80 words
The method of claim 7 further comprising the steps of: cc) low-pass filtering and sub-sampling said image templates obtained in step (a) in order to provide a series of lower resolution image templates; dd) performing an integer position search on the image templates at each resolution level, starting with the lowest resolution level and working up; ee) calculating a transform model at each resolution level in order to predict the positions of the image templates at the next highest level.
References Cited (Backward Citations)
US4924310A
Cited
US5353392A
Cited
US5280530A
Cited
US5566251A
Cited
US5576767A
Cited
US5646679A
Cited
US5649032A
Cited
US5666208A
Cited
US6075895A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-03-03 | 2006-09-02 | 2007-03-04 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-03-03 | 2010-09-02 | 2011-03-04 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-03-03 | 2014-09-02 | 2015-03-04 | $7700.00 | Window Closed / Lapsed |