US6009188A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and system for digital plenoptic imaging
Assignee: Uber Technologies Inc
•
Filing: 1996-03-20
•
Granted: 1999-12-28
•
Expires: 2016-03-20
Abstract
Abstract A computer-based method and system for digital 3-dimensional imaging of an object which allows for viewing images of the object from arbitrary vantage points. The system, referred to as the Lumigraph system, collects a complete appearance of either a synthetic or real object (or a scene), stores a representation of the appearance, and uses the representation to render images of the object from any vantage point. The appearance of an object is a collection of light rays that emanate from the object in all directions. The system stores the representation of the appearance as a set of coefficients of a 4-dimensional function, referred to as the Lumigraph function. From the Lumigraph function with these coefficients, the Lumigraph system can generate 2-dimensional images of the object from any vantage point. The Lumigraph system generates an image by evaluating the Lumigraph function to identify the intensity values of light rays that would emanate from the object to form the image. The Lumigraph system then combines these intensity values to form the image.
Inventors
- 👤 Michael F. Cohen
- 👤 Steven J. Gortler
- 👤 Radek Grzeszczuk
- 👤 Richard S. Szeliski
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-03 20:44
CPC Classifications
G; G06; G06T; G06T15/00; G06T15/50; G06T15/06
Claims Analysis
Total Claims: 34
Claim 1
Independent Claim
295 words
A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from an object, the method comprising: receiving a plurality of pictures of the object, each picture having been taken at a vantage point and having a plurality of pixels, each pixel having an actual initial intensity value, a location, and a direction, the location and direction of each pixel being derived from the vantage point at which the picture is taken; initializing each of the coefficients based on the location and direction of the pixels of the plurality of pictures such that the initialized coefficients represent the actual initial intensity values of the pixels, each coefficient having an initial confidence level indicating confidence that the initialization sets the coefficient to its final value; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, combining coefficients at the next higher level of resolution based on the confidence level of the combined coefficients to generate an averaged coefficient and confidence level for the level of resolution, the averaged coefficient representing an average of the actual initial pixel intensity values represented by the combined coefficients at the next higher level of resolution; and for each of the plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower level of resolution based on the confidence levels at the level of resolution and the next lower level of resolution so that the representation of the actual initial pixel intensity values of the coefficients at the next lower level of resolution is adjusted.
Claim 2
Independent Claim
295 words
A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from an object, the method comprising: receiving a plurality of pictures of the object, each picture having been taken at a vantage point and having a plurality of pixels, each pixel having an actual initial intensity value, a location, and a direction, the location and direction of each pixel being derived from the vantage point at which the picture is taken; initializing each of the coefficients based on the location and direction of the pixels of the plurality of pictures such that the initialized coefficients represent the actual initial intensity values of the pixels, each coefficient having an initial confidence level indicating confidence that the initialization sets the coefficient to its final value; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, combining coefficients at the next higher level of resolution based on the confidence level of the combined coefficients to generate an averaged coefficient and confidence level for the level of resolution, the averaged coefficient representing an average of the actual initial pixel intensity values represented by the combined coefficients at the next higher level of resolution; and for each of the plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower level of resolution based on the confidence levels at the level of resolution and the next lower level of resolution so that the representation of the actual initial pixel intensity values of the coefficients at the next lower level of resolution is adjusted.
Claim 3
Dependent Claim
47 words
The method of claim 1 wherein the plenoptic function is a 4-dimensional plenoptic function such that the four dimensions identify a light ray with a certain direction that emanates from the object and passes through a point on a surface of a convex volume surrounding the object.
Claim 4
Dependent Claim
44 words
The method of claim 1 wherein the plenoptic function is a 5-dimensional plenoptic function such that the five dimensions identify a light ray with a certain direction that emanates from the object and passes through a point outside a convex volume surrounding the object.
Claim 5
Dependent Claim
20 words
The method of claim 1 wherein the pictures are taken with a camera at a plurality of arbitrary vantage points.
Claim 6
Dependent Claim
17 words
The method of claim 4 including determining the vantage point of the picture from the picture itself.
Claim 7
Dependent Claim
19 words
The method of claim 4 including the step of determining the vantage point from information external to the picture.
Claim 8
Dependent Claim
20 words
The method of claim 1 wherein the pictures are taken with a camera at a plurality of pre-defined vantage points.
Claim 9
Dependent Claim
33 words
The method of claim 1 wherein before taking the pictures, the object is placed on a stage with background markers for use in identifying the vantage points at which a picture is taken.
Claim 10
Dependent Claim
56 words
The method of claim 1 wherein each coefficient corresponds to an intensity value associated with an area on a surface of a convex volume surrounding the object and with a direction such that the intensity value represents an average intensity of light rays that emanate from the object and pass through the area in the direction.
Claim 11
Dependent Claim
26 words
The method of claim 1 including the step of rendering an image of the object by evaluating the plenoptic function for each pixel of the image.
Claim 12
Dependent Claim
18 words
The method of claim 1 wherein the step of combining combines coefficients according to a constant basis function.
Claim 13
Dependent Claim
18 words
The method of claim 1 wherein the step of combining combines coefficient according to a quadralinear basis function.
Claim 14
Independent Claim
181 words
A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from a physical object external to the computer system, the method comprising: collecting a plurality of pictures of the physical object, each picture having a vantage point; initializing each of the coefficients based on the collected plurality of pictures, each coefficient having an initial confidence level indicating the level of confidence that the initialization of the coefficient is accurate; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level, combining coefficients at the next higher resolution level based on the confidence levels of each coefficient to generate an averaged coefficient and confidence level for the level; and for each of the plurality of levels of resolution of the coefficients, for catch of a plurality of coefficients at the level, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower resolution level based on the confidence levels at the level and the next lower resolution level.
Claim 15
Independent Claim
181 words
A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from a physical object external to the computer system, the method comprising: collecting a plurality of pictures of the physical object, each picture having a vantage point; initializing each of the coefficients based on the collected plurality of pictures, each coefficient having an initial confidence level indicating the level of confidence that the initialization of the coefficient is accurate; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level, combining coefficients at the next higher resolution level based on the confidence levels of each coefficient to generate an averaged coefficient and confidence level for the level; and for each of the plurality of levels of resolution of the coefficients, for catch of a plurality of coefficients at the level, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower resolution level based on the confidence levels at the level and the next lower resolution level.
Claim 16
Independent Claim
228 words
A method in a computer system for establishing intensity values for calculated light rays that represent an appearance of a physical object external to the computer system, each calculated light ray having a location and direction, the method comprising: collecting a plurality of pictures of the object, each picture having been taken at a vantage point and having pixels, each pixel representing the intensity value of a collected light ray with a location and direction, the location and direction based on the vantage point at which the picture is taken; and for each calculated light ray, determining which collected light rays of which pictures contribute to the intensity value of the calculated light ray based on the location and direction of the collected light rays and on the location and direction of the calculated light ray; and setting the intensity value or the calculated light ray to a combination of the intensity values of the determined collected light rays, wherein intensity value x ij ,p,q for the calculated ray (s i , t j , u p , v q )is: ##EQU24## where is a collection of all collected light rays, where x s ,t,u,v is the intensity value of collected light ray (s, t, u, v), where B i ,j,p,q (s,t,u,v) represents a basis function, and where w i ,j,p,q is a confidence level represented by: ##EQU25##
Claim 17
Independent Claim
228 words
A method in a computer system for establishing intensity values for calculated light rays that represent an appearance of a physical object external to the computer system, each calculated light ray having a location and direction, the method comprising: collecting a plurality of pictures of the object, each picture having been taken at a vantage point and having pixels, each pixel representing the intensity value of a collected light ray with a location and direction, the location and direction based on the vantage point at which the picture is taken; and for each calculated light ray, determining which collected light rays of which pictures contribute to the intensity value of the calculated light ray based on the location and direction of the collected light rays and on the location and direction of the calculated light ray; and setting the intensity value or the calculated light ray to a combination of the intensity values of the determined collected light rays, wherein intensity value x ij ,p,q for the calculated ray (s i , t j , u p , v q )is: ##EQU24## where is a collection of all collected light rays, where x s ,t,u,v is the intensity value of collected light ray (s, t, u, v), where B i ,j,p,q (s,t,u,v) represents a basis function, and where w i ,j,p,q is a confidence level represented by: ##EQU25##
Claim 18
Dependent Claim
14 words
The method of claim 14 wherein the basis function is a constant basis function.
Claim 19
Dependent Claim
10 words
The method of claim 14 wherein the basis function is:
Claim 20
Dependent Claim
14 words
The method of claim 14 wherein the basis function is a quadralinear basis function.
Claim 21
Dependent Claim
36 words
The method of claim 14 wherein the basis function is: where h st is the distance between neighboring locations of calculated light rays and h uv is the distance between neighboring directions of calculated light rays.
Claim 22
Independent Claim
186 words
A method in a computer system for establishing an intensity value for calculated light rays that represent an appearance of a physical object external to the computer system, comprising: providing a plurality of actual light rays representing the appearance, each actual light ray having an intensity value, location, and direction; and for each calculated light ray, determining which actual light rays contribute to the intensity value of the calculated list ray based on the location and direction of the actual light rays and on the location and direction of the calculated light ray; and setting the intensity value of the calculated light ray to a combination of the intensity values of the determined actual light rays, wherein the intensity value x i ,j,p,q for the calculated ray (s i t j , u p , v q ) is: ##EQU26## where is a collection of all collected light rays, where x s ,t,u,v the intensity value of an actual light ray (s, t, u, v), where B i ,j,p,q (s,t,u,v) represents a basis function, and where w i ,j,p,q is a confidence level represented by: ##EQU27##20.
Claim 23
Independent Claim
186 words
A method in a computer system for establishing an intensity value for calculated light rays that represent an appearance of a physical object external to the computer system, comprising: providing a plurality of actual light rays representing the appearance, each actual light ray having an intensity value, location, and direction; and for each calculated light ray, determining which actual light rays contribute to the intensity value of the calculated list ray based on the location and direction of the actual light rays and on the location and direction of the calculated light ray; and setting the intensity value of the calculated light ray to a combination of the intensity values of the determined actual light rays, wherein the intensity value x i ,j,p,q for the calculated ray (s i t j , u p , v q ) is: ##EQU26## where is a collection of all collected light rays, where x s ,t,u,v the intensity value of an actual light ray (s, t, u, v), where B i ,j,p,q (s,t,u,v) represents a basis function, and where w i ,j,p,q is a confidence level represented by: ##EQU27##20.
Claim 24
Dependent Claim
10 words
The method of claim 19 wherein the basis function is:
Claim 25
Dependent Claim
14 words
The method of claim 19 wherein the basis function is a quadralinear basis function.
Claim 26
Dependent Claim
37 words
The method of claim 19 wherein the basis function is: where h st is the distance between neighboring locations of calculated light rays and where h uv is the distance between neighboring directions of calculated light rays.
Claim 27
Independent Claim
232 words
A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
Claim 28
Independent Claim
232 words
A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
Claim 29
Independent Claim
267 words
A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the calculating of an average intensity level for each group includes calculating average intensity levels for successively larger groups of pixels based on previously calculated average intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
Claim 30
Independent Claim
267 words
A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the calculating of an average intensity level for each group includes calculating average intensity levels for successively larger groups of pixels based on previously calculated average intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
Claim 31
Dependent Claim
235 words
A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
Claim 32
Dependent Claim
235 words
A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
Claim 33
Dependent Claim
271 words
A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the calculating of the combined intensity level for each group includes calculating combined intensity levels for successively larger groups of pixels based on previously calculated combined intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels to successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
Claim 34
Dependent Claim
271 words
A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising: setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the calculating of the combined intensity level for each group includes calculating combined intensity levels for successively larger groups of pixels based on previously calculated combined intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels to successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
References Cited (Backward Citations)
US4831659A
Cited
US4928250A
Cited
US5228098A
Cited
US5467404A
Cited
US5727078A
Cited
US5717782A
Cited
US5850352A
Cited
US5675377A
Cited
US5870502A
Cited
US5905500A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-12-27 | 2003-06-28 | 2003-12-28 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-12-27 | 2007-06-28 | 2007-12-28 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-12-27 | 2011-06-28 | 2011-12-28 | $7700.00 | Window Closed / Lapsed |