Patent US6173087B1 successfully refreshed from Google Patents.
US6173087B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Multi-view image registration with application to mosaicing and lens distortion correction
Assignee: Sarnoff Corp
•
Filing: 1997-11-07
•
Granted: 2001-01-09
•
Expires: 2017-11-07
Abstract
Abstract An embodiment of the invention is a system and process for true multi-image alignment that does not rely on the measurements of a reference image being distortion free. For instance, lens distortion is a common imaging phenomenon. When lens distortion is present, none of the images can be assumed to be ideal. In an embodiment of the invention, all the images are modeled as intensity measurements represented in their respective coordinate systems, each of which is related to a reference coordinate system through an interior camera transformation and an exterior view transformation. Motion parameters determined in accordance with an embodiment of the invention dictate the position of the input frames within the reference frame. A reference coordinate system is used, but not a reference image. Motion parameters are computed to warp all input images to a virtual image mosaic in the reference coordinate system of the reference frame. Each pixel in the virtual image mosaic may be predicted by intensities at corresponding pixel positions from more than one image. The error measure, which is the sum of the variances of predicted pixel intensities at each pixel location summed over the virtual image mosaic, is minimized. The embodiment of the invention advantageously maximally uses information present in all images.
Inventors
- 👤 Rakesh Kumar
- 👤 Harpreet Singh Sawhney
- 👤 James Russell Bergen
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-10 01:35
CPC Classifications
G; G06; G06T; G06T3/00; G06T3/14; G06T3/153; G06T3/40; G06T3/4038; G06V; G06V10/00; G06V10/20; G06V10/24; H; H04; H04N; H04N19/00; H04N19/20; H04N19/23; H04N19/50; H04N19/597; G06V10/10; G06V10/16
Claims Analysis
Total Claims: 33
Claim 1
Independent Claim
136 words
Method for multi-view image registration comprising the steps of: a) receiving a plurality of images; b) establishing an ideal coordinate system; c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of: c1) selecting at least two images from the plurality of images; c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 2
Dependent Claim
136 words
Method for multi-view image registration comprising the steps of: a) receiving a plurality of images; b) establishing an ideal coordinate system; c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of: c1) selecting at least two images from the plurality of images; c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 3
Dependent Claim
26 words
The method of claim 1 wherein the ideal coordinate system is related to the coordinate system of the at least two images by an affine transformation.
Claim 4
Dependent Claim
72 words
The method of claim 1 , wherein step c2) includes the steps of selecting one image from the at least two images; deriving a first set of parameters which define a first parametric transformation that translates the one image to a position in the ideal coordinate system; iteratively deriving at least one further set of parameters that define at least one respective further parametric transformation defining the warping of the one image.
Claim 5
Dependent Claim
105 words
The method of claim 1 , wherein: step c2 includes the steps of: defining a set of parametric motion models each with a particular respective functional form and number of parameters; deriving a parametric motion description using an initial selection from said set of models by estimating the corresponding parameters; and iteratively deriving the parameters for each respective further parametric motion description using further selections from said set of models the derivation of the parameters for each further parametric motion description using the parameters derived for each previous parametric motion description to provide an initial estimate of the parameters for the further parametric motion description.
Claim 6
Dependent Claim
80 words
The method of claim 4 , wherein the step of deriving the parametric motion description using the initial selection from the set of models uses a first portion of each image, said first portion having a predetermined size, and the step of iteratively deriving parameters for each respective further parametric motion description uses a further portion of each image, said further portions having a size greater than or equal to the size of the portion used in the previous iteration.
Claim 7
Dependent Claim
99 words
The method of claim 1 , wherein: the step c2) includes the step of: deriving a parametric motion description including a number of initial parameters; and the step c3) includes the steps of: iteratively deriving respective further parametric motion descriptions each further parametric motion description being defined by an equal or greater number of further motion parameters than are included in any previously derived parametric motion description, using the previously derived parametric motion descriptions to provide an initial estimate of the further motion parameters; and deriving image distortion parameters using the further motion parameters derived in a final iteration.
Claim 8
Dependent Claim
61 words
The method of claim 6 , wherein the step of deriving the initial motion parameters uses a first portion of each image, said first portion having a predetermined size, and each subsequent iteration uses a further portion of each image, said further portions having a size greater than or equal to the size of the portion used in a previous iteration.
Claim 9
Dependent Claim
67 words
The method of claim 3 , wherein: the step of deriving the first set of parameters includes the step of deriving global translation parameters; and the step of iteratively deriving at least one further set of parameters includes the steps of: deriving a set of affine parameters using the global translation parameters; and deriving projective parameters and global lens distortion parameters using the set of affine parameters.
Claim 10
Dependent Claim
61 words
The method of claim 8 , wherein the step of deriving the global translation parameters uses approximately the inner ⅓ rectangle of each image; the step of deriving the set of affine parameters uses approximately the inner ⅔ rectangle of each image; and the step of deriving the projective parameters and the global lens distortion parameters uses approximately each entire image.
Claim 11
Dependent Claim
160 words
A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps of: a) receiving, in digital form, a plurality of images; b) establishing an ideal coordinate system; c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of: c1) selecting at least two images from the plurality of images; c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 12
Dependent Claim
160 words
A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps of: a) receiving, in digital form, a plurality of images; b) establishing an ideal coordinate system; c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of: c1) selecting at least two images from the plurality of images; c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 13
Dependent Claim
84 words
The computer-readable medium of claim 10 , wherein: in the step c2), the plurality of instructions cause the processor to perform the step of deriving a parametric motion description including a number of initial parameters; and in step c3), the plurality of instructions cause the processor to perform the step of iteratively deriving respective further parametric motion descriptions each including an equal or greater number of motion parameters than are included in any previously derived parametric motion description, using previously derived parametric motion descriptions.
Claim 14
Dependent Claim
39 words
The computer-readable medium of claim 11 , wherein in step c3) the plurality of instructions cause the processor to further perform the step of deriving image distortion parameters using the further set of parameters derived in a final iteration.
Claim 15
Dependent Claim
73 words
The computer-readable medium of claim 11 , wherein the plurality of instructions cause the processor to perform the step of deriving the initial motion parameters uses a first portion of each image, said first portion having a predetermined size, and each subsequent iteration uses a further portion of each image, each of said further portions having a size greater than or equal to the size of the portion used in the previous iteration.
Claim 16
Dependent Claim
82 words
The computer-readable medium of claim 10 , wherein: the plurality of instructions which cause the processor to perform the step c2) cause the processor to perform the steps of: solving for global translation parameters; and solving for a set of affine parameters using the global translation parameters; and plurality of instructions which cause the processor to perform step c3) cause the processor to perform the step of: solving for projective parameters and global lens distortion parameters using the set of affine parameters.
Claim 17
Dependent Claim
138 words
Apparatus for multi-view image registration comprising: input means for receiving a plurality of images; means for establishing an ideal coordinate system; processing means for processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, comprising: means for selecting at least two images from the plurality of images; means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 18
Dependent Claim
138 words
Apparatus for multi-view image registration comprising: input means for receiving a plurality of images; means for establishing an ideal coordinate system; processing means for processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, comprising: means for selecting at least two images from the plurality of images; means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
Claim 19
Dependent Claim
82 words
The apparatus of claim 15 , wherein the means for computing a plurality of motion parameters includes: means for selecting one image from the at least two images; means for deriving a first set of parameters which define a first parametric transformation that translates the one image to a position in the ideal coordinate system; and means for deriving at least one further set of parameters which define at least one respective further parametric transformation defining a warping of the one image.
Claim 20
Dependent Claim
81 words
The apparatus of claim 15 , wherein: the means for computing the plurality of motion parameters includes means for deriving a parametric motion description comprising a number of initial parameters; and the means for iteratively deriving at least one further set of parameters includes means for iteratively deriving respective further parametric motion descriptions each motion description including an equal or greater numbers of motion parameters than are included in any previously derived motion description, using the previously derived parametric motion descriptions.
Claim 21
Dependent Claim
23 words
The apparatus of claim 17 , further comprising: means for deriving image distortion parameters using the motion parameters derived in a final iteration.
Claim 22
Dependent Claim
80 words
The apparatus of claim 17 , wherein: the means for deriving the parametric motion description comprising the number of initial parameters uses a first portion of each image, said first portion having a predetermined size, and the means for iteratively deriving the respective further parametric motion descriptions uses a respective further portion of each image for each iteration, said further portions each having a size greater than or equal to the size of the portion used in the previous iteration.
Claim 23
Independent Claim
152 words
A method for generating a two-dimensional output mosaic image comprising the steps of: a) receiving a plurality of images; b) generating a set of alignment parameters which map the pixels of the plurality of images to a single coordinate system, including the steps of: b1) selecting at least two images from the plurality of images; b2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and b3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and c) performing two dimensional blending, responsive to the alignment parameters, to combine the plurality of images into the output mosaic image.
Claim 24
Independent Claim
152 words
A method for generating a two-dimensional output mosaic image comprising the steps of: a) receiving a plurality of images; b) generating a set of alignment parameters which map the pixels of the plurality of images to a single coordinate system, including the steps of: b1) selecting at least two images from the plurality of images; b2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and b3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and c) performing two dimensional blending, responsive to the alignment parameters, to combine the plurality of images into the output mosaic image.
Claim 25
Dependent Claim
28 words
The method set forth in claim 20 , wherein step b2) includes the steps of warping the plurality of images by: performing frame-to-frame alignment; and performing frame-to-mosaic alignment.
Claim 26
Dependent Claim
119 words
The method of claim 20 wherein the step of two dimensional blending comprises of: a) selecting for each pixel position in the mosaic image, one of the plurality of images from which a corresponding pixel will be taken; b) creating for each selected one of the plurality of warped images a working mosaic image comprising the selected warped image in its entirety, together with surrounding regions composed of other warped images as determined in step a); c) creating Gaussian and Laplacian pyramids for each of said working mosaics; d) copying pixels from each said working mosaic pyramid into an output mosaic pyramid as determined in step a); and e) reconstructing the output mosaic by collapsing said output mosaic pyramid.
Claim 27
Dependent Claim
90 words
The method of claim 22 wherein the process of selecting one of the plurality of images from which the corresponding pixel value will be taken comprises steps of: a) computing, based on alignment parameters relative to the output mosaic coordinate system, mapped locations corresponding to respective central portions of each image in the plurality of images; and b) creating a Voronoi tessellation in the output mosaic coordinate system in which the said mapped locations of the centers of the respective plurality of images are sites from which distance is determined.
Claim 28
Dependent Claim
28 words
The method of claim 20 , further comprising steps of estimating lens distortion and correcting each of the plurality of images to compensate for the estimated lens distortion.
Claim 29
Dependent Claim
163 words
Apparatus for generating a two dimensional output mosaic image comprising: means for receiving a plurality of images; means for generating a set of alignment parameters which map the pixels of the plurality of images to an ideal coordinate system to form a respective plurality of warped images including: means for selecting at least two images from the plurality of images; means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and means for performing two dimensional blending, responsive to the further set of parameters, to combine the plurality of warped images into the output mosaic image.
Claim 30
Dependent Claim
163 words
Apparatus for generating a two dimensional output mosaic image comprising: means for receiving a plurality of images; means for generating a set of alignment parameters which map the pixels of the plurality of images to an ideal coordinate system to form a respective plurality of warped images including: means for selecting at least two images from the plurality of images; means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and means for performing two dimensional blending, responsive to the further set of parameters, to combine the plurality of warped images into the output mosaic image.
Claim 31
Dependent Claim
122 words
The apparatus of claim 25 wherein the means for performing two dimensional blending comprises: means for selecting, for each pixel position in the output mosaic image, one of the plurality of images from which a corresponding pixel value will be taken; means for creating for each selected one of the plurality of warped images a working mosaic image comprising the selected warped image in its entirety, together with surrounding regions composed of other warped images; means for creating Gaussian and Laplacian pyramids for each of said working mosaics; means for copying pixels from each working mosaic pyramid into an output mosaic pyramid as determined by the means for selecting; and means for reconstructing the output mosaic by collapsing said output mosaic pyramid.
Claim 32
Independent Claim
93 words
A method for estimating lens distortion and correcting each of the plurality of images to compensate for the estimated lens distortion comprising the steps of: a) receiving a plurality of images containing lens distortion; b) selecting a parametric motion description to align said plurality of images to a single coordinate system in the absence of lens distortion; c) estimating said parametric motion descriptions for each of said images while simultaneously estimating lens distortion parameters; and d) warping each of said images using the estimated lens distortion parameters to correct for estimated lens distortion.
Claim 33
Independent Claim
93 words
A method for estimating lens distortion and correcting each of the plurality of images to compensate for the estimated lens distortion comprising the steps of: a) receiving a plurality of images containing lens distortion; b) selecting a parametric motion description to align said plurality of images to a single coordinate system in the absence of lens distortion; c) estimating said parametric motion descriptions for each of said images while simultaneously estimating lens distortion parameters; and d) warping each of said images using the estimated lens distortion parameters to correct for estimated lens distortion.
References Cited (Backward Citations)
US4644582A
Cited
US5187754A
Cited
US5566251A
Cited
US5657402A
Cited
US5768439A
Cited
US5649032A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2004-01-09 | 2004-07-10 | 2005-01-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2008-01-09 | 2008-07-10 | 2009-01-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2012-01-09 | 2012-07-10 | 2013-01-09 | $7700.00 | Window Closed / Lapsed |