US5530774A — Generation of depth image through interpolation a…

Claim 1

Independent Claim 133 words

A method of generating a depth image of a scene comprising the steps of: (a) providing first and second digital images of said scene derived at first and second scene observation locations spatially spaced apart from one another; (b) for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said first and second scene observation locations and associated with respective missing images of said depth image, generating missing angular digital images of said scene in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said first and second digital images; and (c) interlacing said missing angular digital images of said scene and said first and second digital images to derive said depth image.

Claim 2

Independent Claim 133 words

A method of generating a depth image of a scene comprising the steps of: (a) providing first and second digital images of said scene derived at first and second scene observation locations spatially spaced apart from one another; (b) for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said first and second scene observation locations and associated with respective missing images of said depth image, generating missing angular digital images of said scene in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said first and second digital images; and (c) interlacing said missing angular digital images of said scene and said first and second digital images to derive said depth image.

Claim 3

Dependent Claim 81 words

A method according to claim 1, wherein step (b) comprises deriving a respective missing angular digital image of said scene, associated with one of said plurality of spatially separated scene observation locations, by either digitally interpolating or digitally extrapolating said first and second digital images, in dependence upon with whether or not the observation location of said missing angular digital image is spatially between said first and second image observation locations or spatially outside said first and second image observation locations.

Claim 4

Dependent Claim 87 words

A method according to claim 1, wherein step (a) comprises capturing first and second images, from which said first and second digital images are provided, at respective left and right viewing locations t L and t R , that are located in a common image observation plane and are spaced apart from one another along a first axis of a rectangular image viewing coordinate system, where a second axis thereof corresponds to depth into the image, and a third axis thereof is coplanar with said first axis.

Claim 5

Dependent Claim 28 words

A method according to claim 3, wherein respective ones t i of said plurality of spatially separated scene observation locations are located in said common image observation plane.

Claim 6

Dependent Claim 101 words

A method according to claim 4, wherein step (b) comprises generating a respective missing angular digital image associated with a respective one t i of said scene observation locations by deriving a prescribed correspondence between said respective missing angular digital image and said first and second digital images in accordance with a respective intermediate disparity vector field, said respective intermediate disparity vector field having a constant component along said third axis representative of alignment of said missing image with said first and second images, and a variable component along said first axis representative of depths of corresponding points in said scene.

Claim 7

Dependent Claim 93 words

A method according to claim 3, wherein said first and second digital images are digitized into first and second multi-banded digital images DI L and DI R , respectively, having a fixed number of bands, and wherein the band of each of said first and second multi-banded digital images DI L and DI R is comprised of a two-dimensional array of pixels, arrayed in said common image observation plane and having associated pixel values which represent prescribed image parameters taken from a two-dimensional array of points Ω distributed throughout its associated captured image.

Claim 8

Dependent Claim 147 words

A method according to claim 6, wherein said first digital image comprises a plurality of first bands I L ,1, . . . , I L ,B, and said second digital image comprises a plurality of second bands I R ,1, . . . , I R ,B, and wherein step (b) comprises generating a plurality of disparity vector field estimates (U k , V k ), where k=1, 2, . . . , L-1, L+1, . . . , R-1, R+1, . . . , K, in accordance with partial derivatives of filtered image bands for each image band I L ,1, . . . , I L ,B, I R ,1, . . . , I R ,B and a plurality of filters φ 1 , . . . , φ F , through which each band of said first and second images are filtered.

Claim 9

Dependent Claim 57 words

A method according to claim 7, wherein step (b) comprises initially setting a coarsest spatial resolution level disparity vector field estimate of a respective missing image to a preselected constant, and thereafter iteratively modifying said coarsest spatial resolution level disparity vector field estimate of said respective missing image until final estimates of disparity vector fields are obtained.

Claim 10

Dependent Claim 71 words

A method according to claim 8, wherein step (b) comprises iteratively modifying a disparity vector field estimate of said respective missing image in accordance with a multi-level spatial resolution pyramid, wherein, at each increasingly finer resolution pyramid level, filtered images, their associated spatial partial derivatives and estimates of disparity vector fields corresponding to missing images are defined on subgrids of digital image observation points at increasingly finer degrees of spatial resolution.

Claim 11

Dependent Claim 80 words

A method according to claim 9, wherein, in step (b), at a given spatial resolution level, disparity vector fields associated with missing images are defined on a matrix of points Ω of an image plane comprised of N rows of pixels in a first direction corresponding to said first axis by M columns of pixels in a third direction corresponding to said third axis, and wherein said spatial partial derivatives are taken with respect to said third and first directions.

Claim 12

Dependent Claim 64 words

A method according to claim 10, wherein, in step (b), bands of filtered first images, spatial partial derivatives of said bands of filtered first images, bands of filtered second images, and spatial partial derivatives of bands of filtered second images are defined on a common matrix of points Ω' in an image plane having a size of N' rows by M' columns of pixels.

Claim 13

Dependent Claim 50 words

A method according to claim 9, wherein, in step (b), disparity vector fields corresponding to missing images at a respective spatial resolution level are defined by a system of non-linear equations expressed in terms of the respective direction partial derivative variables taken with respect to said third and first axes.

Claim 14

Dependent Claim 113 words

A method according to claim 12, wherein, in step (b), for a respective index gεG specifying a prescribed filter and a particular image band, spatial partial derivatives of a prescribed optical flow function g t are formed with respect to components of a current estimate of the disparity vector along said third and first axes, corresponding to an index g as defined on a grid Ω sum, scaled by a prescribed factor, representative of the ratio of the difference between the observation point of a missing image of interest and said first and second scene observation locations spatially separation ratio factor, of spatial partial derivatives of estimated first and second filtered image bands.

Claim 15

Dependent Claim 47 words

A method according to claim 13, wherein said prescribed optical flow function g t is defined, relative to the matrix of viewpoint locations Ω, as the difference between an estimated left filtered image band and an estimated right filtered image band, as viewed along said first axis.

Claim 16

Dependent Claim 64 words

A method according to claim 14, wherein, in step (b), bands of filtered first images, spatial partial derivatives of said bands of filtered first images, bands of filtered second images, and spatial partial derivatives of bands of filtered second images are defined on a common matrix of points Ω' in an image plane having a size of N' rows by M' columns of pixels.

Claim 17

Dependent Claim 67 words

A method according to claim 15, wherein said estimated left and right filtered image bands are defined relative to respective arrays of image points taken from said first and second scene observation locations, respectively, having associated perspective projections on said common image observation plane estimated from a respective missing image viewpoint, said right and left filtered images being interpolated from said common matrix of image points Ω'.

Claim 18

Dependent Claim 45 words

A method according to claim 16, wherein said respective arrays of image points are defined by respective shifts of each image point of matrix of viewpoint locations Ω, by an amount corresponding to scaling factors of the estimated disparity vector defined at that grid point.

Claim 19

Dependent Claim 118 words

A method according to claim 17, wherein step (b) comprises, for each vector s from a set S specifying a particular direction in said common image observation plane, forming a directional smoothness function (s,∇u), for a variable component u of the current estimate of the disparity vector field along said first axis, corresponding to the missing image, as a finite difference approximation to the directional derivative of said variable component u of the estimate of the disparity vector field corresponding to the missing image along said first axis, where s denotes a vector on the image plane specifying one of a plurality of different directions in said common image observation plane, and S denotes a set of vectors.

Claim 20

Dependent Claim 47 words

A method according to claim 18, wherein said directional smoothness function (s,∇u) is defined on a rectangular subgrid Ω S of said matrix of viewpoint locations Ω, and wherein each grid point of said rectangular subgrid Ω S has a nearest neighbor on the rectangular grid Ω.

Claim 21

Dependent Claim 87 words

A method according to claim 19, wherein, at each grid point of the rectangular subgrid Ω s , said directional smoothness function (s, ∇u) corresponds to the difference between the value of the component u of the current estimate of the disparity vector field corresponding to the missing image at the nearest neighbor, in the direction s, of said grid point and the value of the horizontal component u of the current estimate of the disparity vector field corresponding to the missing image at said grid point.

Claim 22

Dependent Claim 83 words

A method according to claim 18, wherein step (b) comprises forming a system of nonlinear equations relative to the unknown current estimate of the disparity vector field corresponding to the missing image by combining said optical flow function g t and associated spatial partial derivatives g tu , g tv , taken along said first and third axes, respectively, for each filter and each image band specified by said index g εG, together with said directional smoothness function (s,∇u) for each image direction.

Claim 23

Dependent Claim 78 words

A method according to claim 7, wherein step (b) comprises forming a system of nonlinear equations relative to the unknown current estimate of the disparity vector field corresponding to the missing image, linearizing said system of nonlinear equations from which initial estimates of the disparity vector fields are derived relative to the current estimates of the disparity vector fields, and employing solutions to the resulting set of linear equations to modify current estimates of said disparity vector fields.

Claim 24

Dependent Claim 73 words

A method according to claim 22, wherein step (b) further comprises modifying said estimates of disparity vector fields to a prescribed degree of spatial resolution and error and estimating missing images at a plurality of prescribed finest spatial resolution viewpoints, and wherein step (c) comprises interlacing said missing images at said plurality of prescribed finest spatial resolution viewpoints of said scene and said first and second digital images to derive said depth image.

Claim 25

Independent Claim 123 words

A method of generating a depth image of a scene comprising the steps of: (a) generating respective left and right images of said scene derived at stereo scene observation locations spaced apart from one another; (b) for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said stereo scene observation locations, and associated with respective missing images of said depth image, generating missing digital images of said scene in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said left and right images; and (c) interlacing said missing images of said scene and said left and right images to derive said depth image.

Claim 26

Independent Claim 123 words

A method of generating a depth image of a scene comprising the steps of: (a) generating respective left and right images of said scene derived at stereo scene observation locations spaced apart from one another; (b) for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said stereo scene observation locations, and associated with respective missing images of said depth image, generating missing digital images of said scene in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said left and right images; and (c) interlacing said missing images of said scene and said left and right images to derive said depth image.

Claim 27

Dependent Claim 75 words

A method according to claim 24, wherein step (b) comprises deriving a respective missing image of said scene, associated with one of said plurality of spatially separated scene observation locations, by the interpolation or extrapolation of said left and right images, in dependence upon with whether or not the observation location of said missing image is spatially between said left and right image observation locations or spatially outside said left and right image observation locations.

Claim 28

Dependent Claim 82 words

A method according to claim 24, wherein step (a) comprises capturing said left and right images at respective left and right viewing locations t L and t R , that are located in a common image observation plane and are spaced apart from one another along a first, horizontal axis of a rectangular image viewing coordinate system, where a second, depth axis thereof corresponds to depth into the image, and a third, vertical axis thereof is coplanar with said first, horizontal axis.

Claim 29

Dependent Claim 28 words

A method according to claim 26, wherein respective ones t i of said plurality of spatially separated scene observation locations are located in said common image observation plane.

Claim 30

Dependent Claim 96 words

A method according to claim 27, wherein step (b) comprises generating a respective missing image associated with a respective one t i of said scene observation locations by deriving a prescribed correspondence between said respective missing image and said left and right images in accordance with a respective intermediate disparity vector field, said respective intermediate disparity vector field having a constant component along said vertical axis representative of alignment of said missing image with said left and right images, and a variable component along said horizontal axis representative of depths of corresponding points in said scene.

Claim 31

Dependent Claim 92 words

A method according to claim 28, wherein said left and right images are digitized into left and right multi-banded digital images DI L and DI R , respectively, having a fixed number of bands, and wherein the band of each of said left and right multi-banded digital images DI L and DI R is comprised of a two-dimensional array of pixels, arrayed in said common image observation plane and having associated pixel values which represent prescribed image parameters taken from a two-dimensional array of points Ω distributed throughout its associated captured image.

Claim 32

Dependent Claim 151 words

A method according to claim 29, wherein said left multi-banded digital image comprises a plurality of left bands I L .,1, . . . , I L ,B, and said right multi-banded digital image comprises a plurality of right bands I R ,1, . . . , I R ,B, and wherein step (b) comprises generating a plurality of disparity vector field estimates (U k , V k ), where k=1, 2, . . . , L-1, L+1, . . . , R-1, R+1, . . . , K, in accordance with partial derivatives of filtered image bands for each image band I L ,1, . . . , I L ,B, I R ,1, . . . , I R ,B and a plurality of filters φ 1 , . . . , φ F , through which each band of said left and right multi-banded digital images are filtered.

Claim 33

Dependent Claim 79 words

A method according to claim 24, wherein step (b) comprises forming a system of nonlinear relationships with regard to an unknown current estimate of the disparity vector field corresponding to the missing image, linearizing said system of nonlinear relationships from which initial estimates of the disparity vector fields are derived relative to the current estimates of the disparity vector fields, and employing solutions to the resulting set of linear relationships to modify current estimates of said disparity vector fields.

Claim 34

Dependent Claim 73 words

A method according to claim 31, wherein step (b) further comprises modifying said estimates of disparity vector fields to a prescribed degree of spatial resolution and error and estimating missing images at a plurality of prescribed finest spatial resolution viewpoints, and wherein step (c) comprises interlacing said missing images at said plurality of prescribed finest spatial resolution viewpoints of said scene and said left and right digital images to derive said depth image.

Claim 35

Dependent Claim 57 words

A method according to claim 32 wherein step (b) comprises initially setting a coarsest spatial resolution level disparity vector field estimate of a respective missing image to a preselected constant, and thereafter iteratively modifying said coarsest spatial resolution level disparity vector field estimate of said respective missing image until final estimates of disparity vector fields are obtained.

Claim 36

Dependent Claim 71 words

A method according to claim 24 wherein step (b) comprises iteratively modifying a disparity vector field estimate of said respective missing images in accordance with a multi-level spatial resolution pyramid, wherein, at each increasingly finer resolution pyramid level, filtered images, their associated spatial partial derivatives and estimates of disparity vector fields corresponding to missing images are defined on subgrids of digital image observation points at increasingly finer degrees of spatial resolution.

Claim 37

Dependent Claim 80 words

A method according to claim 34, wherein, in step (b), at a given spatial resolution level, disparity vector fields associated with missing images are defined on a matrix of points of an image plane comprised of plural rows of pixels arranged in a horizontal direction along a horizontal axis, and by multiple columns of pixels arranged in a vertical direction along a vertical axis, and wherein said partial spatial derivatives are taken with respect to said vertical and horizontal directions.

Claim 38

Dependent Claim 50 words

A method according to claim 35, wherein, in step (b), disparity vector fields corresponding to missing images at a respective spatial resolution level are defined by a system of non-linear equations expressed in terms of the respective direction partial derivative variables taken with respect to said vertical and horizontal axes.

Claim 39

Dependent Claim 107 words

A method according to claim 36, wherein, in step (b), for a respective index specifying a prescribed filter and a particular image band, spatial partial derivatives of a prescribed optical flow function are formed with respect to components of a current estimate of the disparity vector along said vertical and horizontal directions, corresponding to an index defined on a grid sum, scaled by a prescribed factor, representative of the ratio of the difference between the observation point of a missing image of interest and said left and right scene observation locations spatially separation ratio factor, of spatial partial derivatives of estimated left and right filtered image bands.

Claim 40

Dependent Claim 49 words

A method according to claim 37, wherein said prescribed optical flow function is defined, relative to said matrix of points of said image plane, in accordance with the difference between an estimated left filtered image band and an estimated right filtered image band, as viewed along said horizontal axis.

Claim 41

Independent Claim 188 words

A system for generating a depth image of a scene comprising: an image capture apparatus which is operative to capture respective left and right images of said scene at stereo scene observation locations that are spaced apart from one another in a common image plane and to output respective left and right digital images, each of said left and right digital images being comprised of a matrix of multiband pixels encoded to a prescribed digital code width; and a digital image processor, which is coupled to receive said left and right digital images and generates, for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said stereo scene observation locations, and associated with respective missing images of said depth image, missing digital images of said scene, by processing said left and right digital images in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said left and right images, and interlaces said missing images of said scene and said left and right images to produce said depth image.

Claim 42

Independent Claim 188 words

A system for generating a depth image of a scene comprising: an image capture apparatus which is operative to capture respective left and right images of said scene at stereo scene observation locations that are spaced apart from one another in a common image plane and to output respective left and right digital images, each of said left and right digital images being comprised of a matrix of multiband pixels encoded to a prescribed digital code width; and a digital image processor, which is coupled to receive said left and right digital images and generates, for a plurality of spatially separated scene observation locations, spaced apart from one another and spaced apart from said stereo scene observation locations, and associated with respective missing images of said depth image, missing digital images of said scene, by processing said left and right digital images in accordance with an iteratively refined set of disparity vector fields that associate said plurality of spatially separated scene observation locations with said left and right images, and interlaces said missing images of said scene and said left and right images to produce said depth image.

Claim 43

Dependent Claim 77 words

A system according to claim 39, wherein said digital image processor is operative to generate a respective missing image of said scene, associated with one of said plurality of spatially separated scene observation locations, by interpolating or extrapolating said left and right images, in dependence upon with whether or not the observation location of said missing image is spatially between said left and right image observation locations or spatially outside said left and right image observation locations.

Claim 44

Dependent Claim 86 words

A system according to claim 39, wherein said image capture apparatus is operative to capture said left and right images at respective left and right viewing locations t L and t R , that are located in a common image observation plane and are spaced apart from one another along a first, horizontal axis of a rectangular image viewing coordinate system, where a second, depth axis thereof corresponds to depth into the image, and a third, vertical axis thereof is coplanar with said first, horizontal axis.

Claim 45

Dependent Claim 28 words

A system according to claim 41, wherein respective ones t i of said plurality of spatially separated scene observation locations are located in said common image observation plane.

Claim 46

Dependent Claim 100 words

A system according to claim 42, wherein said digital image processor is operative to generate a respective missing image associated with a respective one t i of said scene observation locations by deriving a prescribed correspondence between said respective missing image and said left and right images in accordance with a respective intermediate disparity vector field, said respective intermediate disparity vector field having a constant component along said vertical axis representative of alignment of said missing image with said left and right images, and a variable component along said horizontal axis representative of depths of corresponding points in said scene.

Claim 47

Dependent Claim 92 words

A system according to claim 43, wherein said left and right images are digitized into left and right multi-banded digital images DI L and DI R , respectively, having a fixed number of bands, and wherein the band of each of said left and right multi-banded digital images DI L and DI R is comprised of a two-dimensional array of pixels, arrayed in said common image observation plane and having associated pixel values which represent prescribed image parameters taken from a two-dimensional array of points Ω distributed throughout its associated captured image.

Claim 48

Dependent Claim 148 words

A system according to claim 44, wherein said left digital image comprises a plurality of left bands I L ,1, . . . , I L ,B, and said right digital image comprises a plurality of right bands I R ,1, . . . , I R ,B, and wherein step (b) comprises generating a plurality of disparity vector field estimates (U k , V k ), where k=1, 2, . . . , L1, L+1, . . . , R-1, R+1, . . . , K, in accordance with partial derivatives of filtered image bands for each image band I L ,1, . . . , I L ,B, I R ,1, . . . , I R ,B and a plurality of filters φ 1 , . . . , φ F , through which each band of said left and right digital images are filtered.

Claim 49

Dependent Claim 85 words

A system according to claim 39, wherein said digital image processor is operative to generate a system of nonlinear equations relative to an unknown current estimate of the disparity vector field corresponding to the missing image, to linearize said system of nonlinear equations from which initial estimates of the disparity vector fields are derived relative to the current estimates of the disparity vector fields, and to modify current estimates of said disparity vector fields in accordance with solutions to the resulting set of linear equations.

Claim 50

Dependent Claim 73 words

A system according to claim 46, wherein said digital image processor is operative to modify said estimates of disparity vector fields to a prescribed degree of spatial resolution and error and estimating missing images at a plurality of prescribed finest spatial resolution viewpoints, and to interlace said missing images at said plurality of prescribed finest spatial resolution viewpoints of said scene with said left and right digital images to derive said depth image.

Claim 51

Dependent Claim 61 words

A system according to claim 47, wherein said digital image processor is operative to initially set a coarsest spatial resolution level disparity vector field estimate of a respective missing image to a preselected constant, and thereafter iteratively modify said coarsest spatial resolution level disparity vector field estimate of said respective missing image until final estimates of disparity vector fields are obtained.

Claim 52

Dependent Claim 76 words

A system according to claim 39, wherein said digital image processor is operative to iteratively modify a disparity vector field estimate of said respective missing images in accordance with a multi-level spatial resolution pyramid, and wherein, at each increasingly finer resolution pyramid level, filtered images, their associated spatial partial derivatives and estimates of disparity vector fields corresponding to missing images are defined on subgrids of digital image observation points at increasingly finer degrees of spatial resolution.

Claim 53

Dependent Claim 83 words

A system according to claim 49, wherein said digital image processor is operative, for a given spatial resolution level, to define disparity vector fields associated with missing images on a matrix of points of an image plane comprised of plural rows of pixels arranged in a horizontal direction along a horizontal axis, and by multiple columns of pixels arranged in a vertical direction along a vertical axis, and wherein said partial spatial derivatives are taken with respect to said vertical and horizontal directions.

Claim 54

Dependent Claim 53 words

A system according to claim 50, wherein said digital image processor is operative to define disparity vector fields corresponding to missing images at a respective spatial resolution level by a system of non-linear equations expressed in terms of the respective direction partial derivative variables taken with respect to said vertical and horizontal axes.

Claim 55

Dependent Claim 110 words

A system according to claim 51, wherein said digital image processor is operative, for a respective index specifying a prescribed filter and a particular image band, to form spatial partial derivatives of a prescribed optical flow function with respect to components of a current estimate of the disparity vector along said vertical and horizontal directions, corresponding to an index defined on a grid sum, scaled by a prescribed factor, representative of the ratio of the difference between the observation point of a missing image of interest and said left and right scene observation locations spatially separation ratio factor, of spatial partial derivatives of estimated left and right filtered image bands.

Claim 56

Dependent Claim 39 words

A system according to claim 52, wherein said prescribed optical flow function is defined in accordance with the difference between an estimated left filtered image band and an estimated right filtered image band, as viewed along said horizontal axis.

Claim 57

Dependent Claim 82 words

The system as set forth in claim 39 wherein said digital image processor further includes electronic means for receiving said left and right digital images and providing, in response thereto, a set of multiband pixel electrical signals representative of said multiband pixels, said missing digital images being in the form of missing digital image electrical signals, said processor being responsive to said multiband pixel electrical signals and said missing digital image electrical signals for providing output signals representative of said depth image.

Annuity Stage	Window Opens	Payment Deadline	Surcharge Period Until	Estimated Fee	Status
3.5 Years (Year 4)	1999-06-25	1999-12-25	2000-06-25	$2000.00	Window Closed / Lapsed
7.5 Years (Year 8)	2003-06-25	2003-12-25	2004-06-25	$3760.00	Window Closed / Lapsed
11.5 Years (Year 12)	2007-06-25	2007-12-25	2008-06-25	$7700.00	Window Closed / Lapsed

Generation of depth image through interpolation and extrapolation of intermediate images derived from stereo image pair using disparity vector fields

Abstract

Inventors

Bibliographic Details

Claims Analysis

References Cited (Backward Citations)

USPTO Annuity & Maintenance Windows

Assigned Categories & Tags

Assign Category Tag