US6754379B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Aligning rectilinear images in 3D through projective registration and calibration
Assignee: Apple Inc
•
Filing: 2003-02-18
•
Granted: 2004-06-22
•
Expires: 2023-02-18
Abstract
Abstract An improved apparatus and method for creating high quality virtual reality panoramas is disclosed that yields dramatic improvements during the authoring and projecting cycles, with speeds up to several orders of magnitude faster than prior systems. In a preferred embodiment, a series of rectilinear images taken from a plurality of rows are pairwise registered with one another, and locally optimized using a pairwise objective function (local error function) that minimizes certain parameters in a projective transformation, using an improved iterative procedure. The local error function values for the pairwise registrations are then saved and used to construct a quadratic surface to approximate a global optimization function (global error function). The chain rule is used to avoid the direct evaluation of the global objective function, saving computation. In one embodiment concerning the blending aspect of the present invention, an improved procedure is described that relies on Laplacian and Gaussian pyramids, using a blend mask whose boundaries are determined by the grassfire transform. An improved iterative procedure is disclosed for the blending that also determines at what level of the pyramid to perform blending, and results in low frequency image components being blended over a wider region and high frequency components being blended over a narrower region. Human interaction and input is also provided to allow manual projective registration, initial calibration and feedback in the selection of photos and convergence of the system.
Inventors
- 👤 Yalin Xiong
- 👤 Ken Turkowski
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:45
CPC Classifications
G; G06; G06T; G06T15/00; G06T15/50; G06T15/506; G06T15/503; G06T3/00; G06T3/40; G06T3/4038; G06T7/00; G06T7/30; G06T7/32; G06V; G06V10/00; G06V10/20; G06V10/24; G06V10/10; G06V10/16
Claims Analysis
Total Claims: 56
Claim 1
Independent Claim
112 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation for a pairwise registration process between at least two overlapping rectilinear images, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two images; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 2
Independent Claim
112 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation for a pairwise registration process between at least two overlapping rectilinear images, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two images; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 3
Dependent Claim
29 words
The method according to claim 1 , wherein determining the vectors comprises determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization.
Claim 4
Dependent Claim
19 words
The method according to claim 1 , wherein the projective transformation comprises a 3×3 matrix having nine projective parameters.
Claim 5
Dependent Claim
62 words
The method according to claim 1 , wherein the pairwise registration process includes a local registration error function between the at least two images, e ij = 1 A ij ∑ overlap ( s ij I j ( X j ) + b ij - I i ( M ij X j ) ) 2 .
Claim 6
Dependent Claim
58 words
The method according to claim 4 , wherein the at least one Hessian matrix, C ij , between the at least two images is determined from the local registration a error function, e ij ( M ij ) ≈e ij 0 +( M ij −M ij 0 ) T C ij ( M ij −M ij 0 ).
Claim 7
Independent Claim
68 words
The method according to 5 , wherein determining the vectors comprises determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization, and wherein a global error function that is optimized during the global optimization process is defined by a function of the sum of the products of overlap area between the at least two images and the local registration error function.
Claim 8
Independent Claim
68 words
The method according to 5 , wherein determining the vectors comprises determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization, and wherein a global error function that is optimized during the global optimization process is defined by a function of the sum of the products of overlap area between the at least two images and the local registration error function.
Claim 9
Dependent Claim
19 words
The method according to claim 1 , wherein the vector of internal parameters comprises a representation of focal length.
Claim 10
Dependent Claim
21 words
The method according to claim 7 , wherein the vector of internal parameters further comprises a representation of image center position.
Claim 11
Dependent Claim
21 words
The method according to claim 8 , wherein the vector of internal parameters further comprises a representation of pixel aspect ratio.
Claim 12
Dependent Claim
19 words
The method according to claim 9 , wherein the vector of internal parameters further comprises a representation of skew.
Claim 13
Dependent Claim
27 words
The method according to claim 1 , wherein the vector of external parameters comprises a representation of camera orientation with respect to a common frame of reference.
Claim 14
Dependent Claim
21 words
The method according to claim 11 , wherein the representation of camera orientation comprises Euler angles including pan, tilt, and roll.
Claim 15
Dependent Claim
21 words
The method according to claim 11 , wherein the representation of camera orientation comprises Euler angles including roll, pitch, and yaw.
Claim 16
Dependent Claim
16 words
The method according to claim 11 , wherein the representation of camera orientation comprises unit quaternions.
Claim 17
Dependent Claim
68 words
The method according to claim 1 , wherein blending the at least two images comprises: for signals in the overlapping region having a frequency higher than a threshold frequency, blending the signals over a first blend region range; and for signals in the overlapping region having a frequency lower than the threshold frequency, blending the signals over a blend region range wider than the first blend region range.
Claim 18
Independent Claim
128 words
A system for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the system comprising: a processor for estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; a global optimizer for determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and a blender for blending the at least two images to provide a smooth transition between the at least two images.
Claim 19
Independent Claim
128 words
A system for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the system comprising: a processor for estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; a global optimizer for determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and a blender for blending the at least two images to provide a smooth transition between the at least two images.
Claim 20
Dependent Claim
19 words
The system according to claim 16 , wherein the projective transformation comprises a 3×3 matrix having nine projective parameters.
Claim 21
Dependent Claim
62 words
The system according to claim 16 , wherein the pairwise registration process includes a local registration error function between the at least two images. e ij = 1 A ij ∑ overlap ( s ij I j ( X j ) + b ij - I i ( M ij X j ) ) 2 .
Claim 22
Dependent Claim
55 words
The system according to claim 18 , wherein the at least one Hessian matrix, C ij , between the at least two images is determined from the local registration error function, e ij ( M ij ) ≈e ij 0 +( M ij −M ij 0 ) T C ij ( M ij 0 ).
Claim 23
Dependent Claim
42 words
The system according to claim 19 , wherein a global error function that is optimized by the global optimizer is defined by a function of the product of overlap area between the at least two images and the local registration error function.
Claim 24
Dependent Claim
19 words
The system according to claim 16 , wherein the vector of internal parameters comprises a representation of focal length.
Claim 25
Dependent Claim
21 words
The system according to claim 21 , wherein the vector of internal parameters further comprises a representation of image center position.
Claim 26
Dependent Claim
21 words
The system according to claim 22 , wherein the vector of internal parameters further comprises a representation of pixel aspect ratio.
Claim 27
Dependent Claim
19 words
The system according to claim 23 , wherein the vector of internal parameters further comprises a representation of skew.
Claim 28
Dependent Claim
27 words
The system according to claim 16 , wherein the vector of external parameters comprises a representation of camera orientation with respect to a common frame of reference.
Claim 29
Dependent Claim
21 words
The system according to claim 25 , wherein the representation of camera orientation comprises Euler angles including pan, tilt, and roll.
Claim 30
Dependent Claim
21 words
The system according to claim 25 , wherein the representation of camera orientation comprises Euler angles including roll, pitch, and yaw.
Claim 31
Dependent Claim
16 words
The system according to claim 25 , wherein the representation of camera orientation comprises unit quaternions.
Claim 32
Dependent Claim
65 words
The system according to claim 16 , wherein: for signals in the overlapping region having a frequency higher than a threshold frequency, the blender blends the signals over a first blend region range; and for signals in the overlapping region having a frequency lower than the threshold frequency, the blender blends the signals over a blend region range wider than the first blend region range.
Claim 33
Independent Claim
118 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing at least one Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three-dimensional geometry surface.
Claim 34
Independent Claim
118 words
A method for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the method comprising: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing at least one Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three-dimensional geometry surface.
Claim 35
Dependent Claim
29 words
The method according to claim 30 , wherein determining the vectors comprises determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization.
Claim 36
Dependent Claim
15 words
The method according to claim 30 , wherein the three-dimensional geometry surface is a sphere.
Claim 37
Dependent Claim
134 words
A computer program product for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the computer program product comprising: a computer-readable medium; and computer program code, encoded on the medium, for: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 38
Dependent Claim
134 words
A computer program product for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the computer program product comprising: a computer-readable medium; and computer program code, encoded on the medium, for: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation relating the images and yielding at least one Hessian matrix describing a registration error landscape in the neighborhood of the projective transformation; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; and blending the at least two images to provide a smooth transition between the at least two images.
Claim 39
Dependent Claim
40 words
The computer program product according to claim 33 , wherein the computer program code for determining the vectors comprises computer program code for determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization.
Claim 40
Dependent Claim
21 words
The computer program product according to claim 33 , wherein the projective transformation comprises a 3×3 matrix having nine projective parameters.
Claim 41
Dependent Claim
64 words
The computer program product according to claim 33 , wherein the pairwise registration process includes a local registration error function between the at least two images. e ij = 1 A ij ∑ overlap ( s ij I j ( X j ) + b ij - I i ( M ij X j ) ) 2 .
Claim 42
Dependent Claim
59 words
The computer program product according to claim 36 , wherein the at least one Hessian matrix, C ij , between the at least two images is determined from the local registration error function, e ij ( M ij ) ≈e ij 0 +( M ij −M ji 0 ) T C ij ( M ij −M ij 0 ).
Claim 43
Dependent Claim
48 words
The computer program product according to claim 37 , wherein a global error function that is optimized during the global optimization process is defined by a function of the sum of the products of overlap area between the at least two images and the local registration error function.
Claim 44
Dependent Claim
21 words
The computer program product according to claim 33 , wherein the vector of internal parameters comprises a representation of focal length.
Claim 45
Dependent Claim
23 words
The computer program product according to claim 39 , wherein the vector of internal parameters further comprises a representation of image center position.
Claim 46
Dependent Claim
23 words
The computer program product according to claim 40 , wherein the vector of internal parameters further comprises a representation of pixel aspect ratio.
Claim 47
Dependent Claim
21 words
The computer program product according to claim 41 , wherein the vector of internal parameters further comprises a representation of skew.
Claim 48
Dependent Claim
29 words
The computer program product according to claim 33 , wherein the vector of external parameters comprises a representation of camera orientation with respect to a common frame of reference.
Claim 49
Dependent Claim
23 words
The computer program product according to claim 43 , wherein the representation of camera orientation comprises Euler angles including pan, tilt and roll.
Claim 50
Dependent Claim
23 words
The computer program product according to claim 43 , wherein the representation of camera orientation comprises Euler angles including roll, pitch, and yaw.
Claim 51
Dependent Claim
18 words
The computer program product according to claim 43 , wherein the representation of camera orientation comprises unit quarternions.
Claim 52
Dependent Claim
79 words
The computer program product according to claim 33 , wherein the computer program code for blending the at least two images comprises computer program code for: for signals in the overlapping region having a frequency higher than a threshold frequency, blending the signals over a first blend region range; and for signals in the overlapping region having a frequency lower than the threshold frequency, blending the signals over a blend region range wider than the first blend region range.
Claim 53
Dependent Claim
134 words
A computer program product for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the computer program product comprising: a computer-readable medium; and computer program code, encoded on the medium, for: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing at least one Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three-dimensional geometry surface.
Claim 54
Dependent Claim
134 words
A computer program product for creating a panoramic image from a plurality of rectilinear images obtained from a camera, the computer program product comprising: a computer-readable medium; and computer program code, encoded on the medium, for: estimating a projective transformation during a pairwise registration process between at least two overlapping rectilinear images obtained from a camera, said projective transformation providing at least one Hessian matrix relating the at least two rectilinear images; determining a vector of internal parameters and a vector of external parameters from the at least one Hessian matrix to minimize image discrepancy in an overlapping region between the at least two overlapping rectilinear images; blending the at least two images to provide a smooth transition between the at least two images; and mapping the blended images onto a three-dimensional geometry surface.
Claim 55
Dependent Claim
40 words
The computer program product according to claim 48 , wherein the computer program code for determining the vectors comprises computer program code for determining the vectors to minimize image discrepancy in the overlapping region during a process of global optimization.
Claim 56
Dependent Claim
17 words
The computer program product according to claim 48 , wherein the three-dimensional geometry surface is a sphere.
References Cited (Backward Citations)
US5572235A
Cited
US5664082A
Cited
US6075905A
Cited
US5986668A
Cited
US6009190A
Cited
US6078701A
Cited
US6097854A
Cited
US5951475A
Cited
US6434265B1
Cited
US6549651B2
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-06-22 | 2007-12-22 | 2008-06-22 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-06-22 | 2011-12-22 | 2012-06-22 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-06-22 | 2015-12-22 | 2016-06-22 | $7700.00 | Window Closed / Lapsed |