US6549651B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Aligning rectilinear images in 3D through projective registration and calibration
Assignee: Apple Inc
•
Filing: 2002-02-14
•
Granted: 2003-04-15
•
Expires: 2022-02-14
Abstract
Abstract An improved apparatus and method for creating high quality virtual reality panoramas is disclosed that yields dramatic improvements during the authoring and projecting cycles, with speeds up to several orders of magnitude faster than prior systems. In a preferred embodiment, a series of rectilinear images taken from a plurality of rows are pairwise registered with one another, and locally optimized using a pairwise objective function (local error function) that minimizes certain parameters in a projective transformation, using an improved iterative procedure. The local error function values for the pairwise registrations are then saved and used to construct a quadratic surface to approximate a global optimization function (global error function). The chain rule is used to avoid the direct evaluation of the global objective function, saving computation. In one embodiment concerning the blending aspect of the present invention, an improved procedure is described that relies on Laplacian and Gaussian pyramids, using a blend mask whose boundaries are determined by the grassfire transform. An improved iterative procedure is disclosed for the blending that also determines at what level of the pyramid to perform blending, and results in low frequency image components being blended over a wider region and high frequency components being blended over a narrower region. Human interaction and input is also provided to allow manual projective registration, initial calibration and feedback in the selection of photos and convergence of the system.
Inventors
- 👤 Yalin Xiong
- 👤 Ken Turkowski
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:18
CPC Classifications
G; G06; G06T; G06T15/00; G06T15/50; G06T15/506; G06T15/503; G06T3/00; G06T3/40; G06T3/4038; G06T7/00; G06T7/30; G06T7/32; G06V; G06V10/00; G06V10/20; G06V10/24; G06V10/10; G06V10/16
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
110 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images during a process of global optimization; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 2
Independent Claim
110 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images during a process of global optimization; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 3
Dependent Claim
19 words
The method according to claim 1 , wherein the projective transformation is a 3×3 matrix comprising 9 projective parameters.
Claim 4
Dependent Claim
62 words
The method according to claim 1 , wherein the pairwise registration process includes a local registration error function between the at least two images, e ij = 1 A ij ∑ overlap ( s ij I j ( X j ) + b ij - I i ( M ij X j ) ) 2 .
Claim 5
Dependent Claim
54 words
The method according to claim 3 , wherein the Hessian matrix, C ij , between the at least two images is determined from the local registration error function, e ij ( M ij )≈ e ij 0 +( M ij −M ij 0 ) T C ij ( M ij −M ij 0 ).
Claim 6
Dependent Claim
23 words
The method according to claim 1 , wherein the vector of internal parameters comprise of image center position, aspect ratio, and focal length.
Claim 7
Dependent Claim
25 words
The method according to claim 1 , wherein the vector of external parameters represents the camera orientation with respect to a common frame of reference.
Claim 8
Dependent Claim
43 words
The method according to claim 4 , wherein a global error function that is optimized during the global optimization process is defined by a function of the product of overlap area between the at least two images and the local registration error function.
Claim 9
Dependent Claim
35 words
The method according to claim 1 , wherein the blending is performed such that high frequency signals in the overlapping region between the at least two images are blended over a narrow blend region range.
Claim 10
Dependent Claim
35 words
The method according to claim 1 , wherein the blending is performed such that low frequency signals in the overlapping region between the at least two images are blended over a wide blend region range.
Claim 11
Independent Claim
114 words
A system for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the system comprising: a processor for estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; a global optimizer for determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and a blender for blending the at least two images to provide a smooth transition between the at least two images.
Claim 12
Independent Claim
114 words
A system for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the system comprising: a processor for estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; a global optimizer for determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and a blender for blending the at least two images to provide a smooth transition between the at least two images.
Claim 13
Dependent Claim
19 words
The system according to claim 10 , wherein the projective transformation is a 3×3 matrix comprising 9 projective parameters.
Claim 14
Dependent Claim
62 words
The system according to claim 10 , wherein the pairwise registration process includes a local registration error function between the at least two images, e ij = 1 A ij ∑ overlap ( s ij I j ( X j ) + b ij - I i ( M ij X j ) ) 2 .
Claim 15
Dependent Claim
62 words
The system according to claim 12 , wherein the Hessian matrix, C ij , between the at least two images is determined from the local registration error function, e ij ( M ij ) ≈ e ij 0 + ( M ij - M ij 0 ) T C ij ( M ij - M ij 0 ) .
Claim 16
Dependent Claim
23 words
The system according to claim 10 , wherein the vector of internal parameters comprise of image center position, aspect ratio, and focal length.
Claim 17
Dependent Claim
25 words
The system according to claim 10 , wherein the vector of external parameters represents the camera orientation with respect to a common frame of reference.
Claim 18
Dependent Claim
42 words
The system according to claim 13 , wherein a global error function that is optimized by the global optimizer is defined by a function of the product of overlap area between the at least two images and the local registration error function.
Claim 19
Dependent Claim
35 words
The system according to claim 10 , wherein the blending is performed such that high frequency signals in the overlapping region between the at least two images are blended over a narrow blend region range.
Claim 20
Dependent Claim
35 words
The system according to claim 10 , wherein the blending is performed such that low frequency signals in the overlapping region between the at least two images are blended over a wide blend region range.
Claim 21
Independent Claim
120 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images during a process of global optimization; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three dimensional geometry surface.
Claim 22
Independent Claim
120 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing a Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images during a process of global optimization; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three dimensional geometry surface.
Claim 23
Dependent Claim
16 words
The method according to claim 19 , wherein the three dimensional geometry surface is a sphere.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-04-14 | 2006-10-14 | 2007-04-15 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-04-14 | 2010-10-14 | 2011-04-15 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-04-14 | 2014-10-14 | 2015-04-15 | $7700.00 | Window Closed / Lapsed |