Method and apparatus for three-dimensional, textured models from plural video images

Claim 3

The method of claim 1 further comprising the steps of: a. segmenting the object into a plurality of bounded, two-dimensional object regions having contours; b. based on the estimated parameters, defining the spatial location and orientation of the object regions; and c. defining the final pixelmaps to possess contours that match those of the object regions.

Claim 4

The method of claim 2 wherein the two-dimensional object regions are quadrilaterals.

Claim 5

The method of claim 2 wherein pixels from each digitized video frame that contains at least part of an object region are mapped to a separate final pixelmap defined with respect to that object region, and further comprising the steps of: a. increasing the sampling density of each such final pixelmap by bilinear interpolation; and b. combining the separate final pixelmaps into a composite final pixelmap corresponding to the object region by averaging.

Claim 6

The method of claim 4 further comprising the step of registering the separate final pixelmaps prior to combining them into a composite final pixelmap.

Claim 7

The method of claim 4 wherein at least some of the registered pixelmaps differ in video resolution and are combined by averaging according to weights derived from differences in video resolution.

Claim 8

The method of claim 1 wherein recursive estimation is carried out using an extended Kalman filter.

Claim 9

The method of claim 1 wherein tracking is performed by performing a normalized correlation on a plurality of image-based templates.

Claim 10

The method of claim 1 further comprising the step of increasing the sampling density of the final pixelmaps by bilinear interpolation.

Claim 11

The method of claim 1 wherein pixels from a plurality of digitized video frames are mapped to pixel locations within a single final pixelmap.

Claim 12

The method of claim 1 wherein each tracked object feature corresponds to a real object feature in a coordinate system wherein a camera having an unknown focal length lies on a camera plane, the tracked object features lie on an image plane whose distance from the camera plane is equal to the unknown focal length, and an unknown depth parameter α i specifies, for each tracked object feature i, the distance between the image plane and the real object feature corresponding to the tracked object feature i; and the unknown parameter specifying camera focal length is the inverse focal length β.

Claim 13

The method of claim 11 wherein the unknown parameters specifying camera translation are t X , t Y and t Z β, wherein t X and t Y correspond to camera motion parallel to the image plane and t Z corresponds to camera motion perpendicular to the image plane; the unknown parameters specifying camera rotation are Euler angles ω x , ω y , ω z , corresponding, respectively, to rotation about the two axes parallel to and the axis perpendicular to the image plane; and the unknown parameters specifying spatial locations of the tracked features are the depth parameters α i .

Claim 14

The method of claim 12 wherein recursive estimation is carried out using an extended Kalman filter that operates on the parameters t X , t Y , t Z β, ω x , ω y , ω z , β, and α i for each object feature.

Claim 17

Apparatus for generating, from digital pixelmaps representative of a multiple-frame video recording of an object, geometrically specified, textured pixelmaps that collectively represent a three-dimensional model of the object, the apparatus comprising: a. electronic storage means; b. means for partitioning the storage means into memory partitions for electronically storing each video-frame pixelmap and a plurality of final pixelmaps; b. means for selecting and electronically identifying at least seven object features from a first digitized video frame; c. means for tracking the features among a plurality of digitized video frames to determine changes in the relative distances therebetween; d. means for recursively estimating, based on changes in the relative distances, a plurality of unknown parameters including camera focal length, camera translation, camera rotation, and spatial locations of the tracked features; e. means for defining, based on the estimated parameters, vectors that specify spatial locations of object details specified by each pixel in a plurality of digitized video frames; and f. means for mapping, based on the spatial vectors, pixels from the digitized video frames that correspond to a single planar object surface to locations within the final pixelmaps, each final pixelmap being representative of a single object surface.

Claim 18

Apparatus for generating, from digital pixelmaps representative of a multiple-frame video recording of an object, geometrically specified, textured pixelmaps that collectively represent a three-dimensional model of the object, the apparatus comprising: a. electronic storage means; b. means for partitioning the storage means into memory partitions for electronically storing each video-frame pixelmap and a plurality of final pixelmaps; b. means for selecting and electronically identifying at least seven object features from a first digitized video frame; c. means for tracking the features among a plurality of digitized video frames to determine changes in the relative distances therebetween; d. means for recursively estimating, based on changes in the relative distances, a plurality of unknown parameters including camera focal length, camera translation, camera rotation, and spatial locations of the tracked features; e. means for defining, based on the estimated parameters, vectors that specify spatial locations of object details specified by each pixel in a plurality of digitized video frames; and f. means for mapping, based on the spatial vectors, pixels from the digitized video frames that correspond to a single planar object surface to locations within the final pixelmaps, each final pixelmap being representative of a single object surface.

Claim 19

The apparatus of claim 15 wherein the mapping means is also configured to: a. segment the object into bounded, two-dimensional object regions having contours; b. define, based on the estimated parameters, the spatial location and orientation of the object regions; and c. define the final pixelmaps to possess contours that match those of the object regions.

Claim 20

The apparatus of claim 16 wherein the two-dimensional object regions are quadrilaterals.

Claim 21

The apparatus of claim 16 wherein the mapping means maps pixels from each digitized video frame that contains at least part of an object region to a separate final pixelmap defined with respect to that object region, and further wherein the mapping means is configured to: a. increase the sampling density of each such final pixelmap by bilinear interpolation; and b. combine the separate final pixelmaps into a composite final pixelmap corresponding to the object region by averaging.

Claim 22

The apparatus of claim 18 wherein the mapping means is further configured to register the separate final pixelmaps prior to combining them into a composite final pixelmap.

Claim 23

The apparatus of claim 19 wherein wherein at least some of the registered pixelmaps differ in video resolution and the mapping means is further configured to combine the registered pixelmaps by averaging according to weights derived from differences in video resolution.

Claim 24

The apparatus of claim 15 wherein each tracked object feature corresponds to a real object feature in a coordinate system wherein a camera having an unknown focal length lies on a camera plane, the tracked object features lie on an image plane whose distance from the camera plane is equal to the unknown focal length, and an unknown depth parameter α i specifies, for each tracked object feature i, the distance between the image plane and the real object feature corresponding to the tracked object feature i; and the unknown parameter specifying camera focal length is the inverse focal length β.

Claim 25

The apparatus of claim 21 wherein the unknown parameters specifying camera translation are t X , t Y and t Z β, wherein t X and t Y correspond to camera motion parallel to the image plane and t Z corresponds to camera motion perpendicular to the image plane; the unknown parameters specifying camera rotation are Euler angles ω x , ω y , ω z , corresponding, respectively, to rotation about the two axes parallel to and the axis perpendicular to the image plane; and the unknown parameters specifying spatial locations of the tracked features are the depth parameters α i .

Claim 26

The apparatus of claim 22 wherein means for recursively estimating operates by means of an extended Kalman filter that operates on the parameters t X , t Y , t Z β, ω x , ω y , ω z , β, and α i for each object feature.

Claim 27

The apparatus of claim 15 wherein the tracking means operates by performing a normalized correlation on a plurality of image-based templates.

Claim 28

The apparatus of claim 15 wherein the mapping means is configured to increase the sampling density of the final pixelmaps by bilinear interpolation.

Claim 29

The apparatus of claim 15 wherein the mapping means maps pixels from a plurality of digitized video frames to pixel locations within a single final pixelmap.