Patent US6522787B1 successfully refreshed from Google Patents.
US6522787B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and system for rendering and combining images to form a synthesized view of a scene containing image information from a second image
Assignee: SRI International Inc
•
Filing: 1997-08-25
•
Granted: 2003-02-18
•
Expires: 2017-08-25
Abstract
Abstract An image processing system for imaging a scene to mosaic, selecting a new viewpoint of the scene, and rendering a synthetic image from the mosaic of the scene from that new viewpoint. The synthesized image is then combined with a second image. The combination of the second image and the synthetic image generates a composite image containing a realistic combination of objects in the second image and the scene. Using this system, a production set or other scene need only be created once, then imaged by the system. Thereafter, through image processing, any view of the scene can be synthesized and combined with separately imaged performers or other objects to generate the composite image. As such, a production set or other scene can be repetitively reused without recreating the physical scene.
Inventors
- 👤 Rakesh Kumar
- 👤 Keith James Hanna
- 👤 James R. Bergen
- 👤 Padmanabhan Anandan
- 👤 Kevin Williams
- 👤 Mike Tinker
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 12:40
CPC Classifications
H; H04; H04N; H04N5/00; H04N5/222; H04N5/2224; G; G06; G06T; G06T15/00; G06T15/10; H04N5/262; H04N5/272
Claims Analysis
Total Claims: 29
Claim 1
Independent Claim
154 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images, said plurality of first images including at least an image having a first viewpoint of said scene and an image having a second viewpoint of said scene, said first and second viewpoints being non-coincidental; generating a second image; producing an indicium of viewpoint while generating said second image; rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and compositing said synthesized image with said second image to produce a composite image including parallax information.
Claim 2
Independent Claim
154 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images, said plurality of first images including at least an image having a first viewpoint of said scene and an image having a second viewpoint of said scene, said first and second viewpoints being non-coincidental; generating a second image; producing an indicium of viewpoint while generating said second image; rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and compositing said synthesized image with said second image to produce a composite image including parallax information.
Claim 3
Dependent Claim
13 words
The method of claim 1 wherein said mosaic is a plurality of mosaics.
Claim 4
Dependent Claim
25 words
The method of claim 1 wherein said step of generating said second image includes the step of rendering said second image from a second mosaic.
Claim 5
Dependent Claim
16 words
The method of claim 1 wherein said indicium of viewpoint is at least one camera parameter.
Claim 6
Dependent Claim
39 words
The method of claim 1 wherein said step of generating a second image further comprises the step of imaging a performance before a background having a color that can be removed from the second image using a chroma-key technique.
Claim 7
Dependent Claim
41 words
The method of claim 1 wherein said step of producing an indicium of viewpoint further comprises the step of recording at least one camera parameter selected from the following group of camera parameters: three-dimensional position, pan, tilt, roll and zoom parameters.
Claim 8
Dependent Claim
51 words
The method of claim 1 wherein said step of producing an indicium of viewpoint further comprising the steps of: providing a plurality of indicia proximate an area where a performance is imaged as said second image; and imaging said plurality of indicia as a landmark image while generating said second image.
Claim 9
Dependent Claim
22 words
The method of claim 7 wherein said landmark imaging step further comprises the step of generating said second image using a camera.
Claim 10
Dependent Claim
25 words
The method of claim 7 further comprising a step of imaging a plurality of reference images representing said indicia viewed from a plurality of viewpoints.
Claim 11
Dependent Claim
21 words
The method of claim 9 further comprising a step of correcting said indicium of viewpoint using said plurality of reference images.
Claim 12
Dependent Claim
42 words
The method of claim 10 further comprising the steps of: measuring an absolute location of the indicia; and computing camera pose in response to the absolute location of the indicia and a measurement of camera pose represented by said indicium of viewpoint.
Claim 13
Dependent Claim
34 words
The method of claim 1 wherein said mosaic is a three-dimensional mosaic containing an image mosaic representing a panoramic view of the scene and a shape mosaic representing a three-dimensional geometry of the scene.
Claim 14
Dependent Claim
15 words
The method of claim 12 wherein said synthesized image is rendered from said three-dimensional mosaic.
Claim 15
Dependent Claim
46 words
The method of claim 13 wherein said mosaic generating step further comprises the steps of: imaging a scene from a plurality of grid points; generating a mosaic at each of said grid points; and storing each of said mosaics with reference to a particular grid point.
Claim 16
Dependent Claim
60 words
The method of claim 14 wherein said step of generating a mosaic at each of said grid points further comprises: generating a plurality of mosaics at each grid point, where each of said mosaics is generated at a different illumination level of the scene; rendering said synthesized image by matching illumination levels of said synthesized image and said second image.
Claim 17
Dependent Claim
18 words
The method of claim 1 wherein the indicium of viewpoint is produced by a user controlled input device.
Claim 18
Independent Claim
143 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene; generating a second image; providing a plurality of indicia proximate an area where a performance is imaged as said second image; imaging, using a tracking camera that is separate from a camera used to generate said second image, said plurality of indicia as a reference image while generating said second image rendering, in response to said indicium of viewpoint; a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and compositing said synthesized image with said second image to produce a composite image.
Claim 19
Independent Claim
143 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene; generating a second image; providing a plurality of indicia proximate an area where a performance is imaged as said second image; imaging, using a tracking camera that is separate from a camera used to generate said second image, said plurality of indicia as a reference image while generating said second image rendering, in response to said indicium of viewpoint; a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and compositing said synthesized image with said second image to produce a composite image.
Claim 20
Independent Claim
219 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene; generating a second image; providing a plurality of indicia proximate an area where a performance is imaged as said second image; imaging said plurality of indicia as a reference image while generating said second image; rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; compositing said synthesized image with said second image to produce a composite image; imaging a plurality of reference images representing said indicia viewed from a plurality of viewpoints; correcting said indicium of viewpoint using said plurality of reference images; selecting a reference image that represents a viewpoint that best represents a viewpoint indicated by said indicium of viewpoint; warping said selected reference image into alignment with a landmark image generated from said viewpoint represented by said indicium of viewpoint; computing, in response to said alignment, offset parameters for correcting said indicium of viewpoint; and generating said synthesized image in response to said corrected indicium of viewpoint.
Claim 21
Independent Claim
219 words
A method for image processing comprising the steps of: generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene; generating a second image; providing a plurality of indicia proximate an area where a performance is imaged as said second image; imaging said plurality of indicia as a reference image while generating said second image; rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; compositing said synthesized image with said second image to produce a composite image; imaging a plurality of reference images representing said indicia viewed from a plurality of viewpoints; correcting said indicium of viewpoint using said plurality of reference images; selecting a reference image that represents a viewpoint that best represents a viewpoint indicated by said indicium of viewpoint; warping said selected reference image into alignment with a landmark image generated from said viewpoint represented by said indicium of viewpoint; computing, in response to said alignment, offset parameters for correcting said indicium of viewpoint; and generating said synthesized image in response to said corrected indicium of viewpoint.
Claim 22
Dependent Claim
142 words
An image processing system comprising: mosaic generation means for generating a mosaic of first images representing a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images and having a first viewpoint; viewpoint indicium means for generating an indicium of viewpoint, non-coincidental with said first viewpoint, for a second image; rendering means, connected to said mosaic generation means and said viewpoint indicium means, for producing a synthetic image derived from said mosaic that represents a view of the scene corresponding to the indicium of viewpoint; and compositing means, connected to said rendering means and said viewpoint indicium means, for combining said synthesized image and said second image to form a composite image including parallax information.
Claim 23
Dependent Claim
142 words
An image processing system comprising: mosaic generation means for generating a mosaic of first images representing a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images and having a first viewpoint; viewpoint indicium means for generating an indicium of viewpoint, non-coincidental with said first viewpoint, for a second image; rendering means, connected to said mosaic generation means and said viewpoint indicium means, for producing a synthetic image derived from said mosaic that represents a view of the scene corresponding to the indicium of viewpoint; and compositing means, connected to said rendering means and said viewpoint indicium means, for combining said synthesized image and said second image to form a composite image including parallax information.
Claim 24
Dependent Claim
38 words
The system of claim 19 wherein said mosaic generation means further comprises: means for generating a plurality of mosaics, where each mosaic in said plurality of mosaics represents a panoramic view of the scene from a unique location.
Claim 25
Dependent Claim
42 words
The system of claim 19 wherein said viewpoint indicium means further comprises: means for tracking a camera pose as said camera produces said second image; and means, connected to said tracking means, for generating said camera pose as an indicium of viewpoint.
Claim 26
Dependent Claim
24 words
The system of claim 19 wherein said viewpoint indicium means further comprises: means for producing said indicium of viewpoint in response to user commands.
Claim 27
Dependent Claim
25 words
The system of claim 19 wherein said compositing means further comprises: means for combining said synthetic image and said second image using a chroma-key process.
Claim 28
Dependent Claim
28 words
The system of claim 21 wherein said camera pose includes indicium of viewpoint selected from the following group of camera parameters: three-dimensional position, pan, tilt, roll and zoom.
Claim 29
Dependent Claim
43 words
The system of claim 19 further comprising: second mosaic generation means for generating a second mosaic of images representing a second scene; and second rendering means, connected to said viewpoint indicium means and said second mosaic generation means, for producing said second image.
References Cited (Backward Citations)
US3233508A
Cited
US3081557A
Cited
US3439105A
Cited
US3580978A
Cited
US4231642A
Cited
US4323920A
Cited
US4393394A
Cited
US4515450A
Cited
US4758892A
Cited
US4797942A
Cited
US5353392A
Cited
US5187571A
Cited
US5187754A
Cited
US5264933A
Cited
US5428543A
Cited
US5259040A
Cited
US5455633A
Cited
US5444478A
Cited
US5424773A
Cited
EP0618471A2
Cited
US5613032A
Cited
US5581637A
Cited
US5703961A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-02-17 | 2006-08-19 | 2007-02-18 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-02-17 | 2010-08-19 | 2011-02-18 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-02-17 | 2014-08-19 | 2015-02-18 | $7700.00 | Window Closed / Lapsed |