Patent US6072496A successfully refreshed from Google Patents.
US6072496A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects
Assignee: Microsoft Technology Licensing LLC
•
Filing: 1998-06-08
•
Granted: 2000-06-06
•
Expires: 2018-06-08
Abstract
Abstract The method captures a 3D model of a face, which includes a 3D mesh and a series of deformations of the mesh that define changes in position of the mesh over time (e.g., for each frame). The method also builds a texture map associated with each frame in an animation sequence. The method achieves significant advantages by using markers on an actor's face to track motion of the face over time and to establish a relationship between the 3D model and texture. Specifically, videos of an actor's face with markers are captured from multiple cameras. Stereo matching is used to derive 3D locations of the markers in each frame. A 3D scan is also performed on the actor's face with the markers to produce an initial mesh with markers. The markers from the 3D scan are matched with the 3D locations of the markers in each frame from the stereo matching process. The method determines how the position of the mesh changes from frame to frame by matching the 3D locations of the markers from one frame to the next. The method derives textures for each frame by removing the dots from the video data, finding a mapping between texture space and the 3D space of the mesh, and combining the camera views for each frame into a signal texture map. The data needed to represent facial animation includes: 1) an initial 3D mesh, 2) 3D deformations of the mesh per frame, and 3) a texture map associated with each deformation. The method compresses 3D geometry by decomposing the deformation data into basis vectors and coefficients. The method compresses the textures using video compression.
Inventors
- 👤 Brian Guenter
- 👤 Cindy Marie Grimm
- 👤 Henrique Sarmento Malvar
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:44
CPC Classifications
G; G06; G06T; G06T13/00; G06T13/20; G06T13/40; G06T7/00; G06T7/50; G06T7/55; G06T7/593; G06V; G06V10/00; G06V10/20; G06V10/24; G06V10/245; G06V20/00; G06V20/60; G06V20/64; G06V40/00; G06V40/10; G06V40/16; G06V40/161; G06V40/166; G06V40/174; G06V40/176; G06T2200/00; G06T2200/08
Claims Analysis
Total Claims: 31
Claim 1
Independent Claim
152 words
A method for capturing a 3D geometric model of an animated 3D object, and color and shading information of a surface of the animated 3D object, the method comprising: capturing a base mesh of the 3D object; capturing n simultaneous video sequences of the animated 3D object with markers applied to the surface of the object from each of n cameras positioned around the 3D object, where n is two or more and the video sequences include n camera images per frame; determining 3D positions of the markers from the n 2D camera images for frames in the video sequence; determining an association between the 3D positions of the markers and vertices in the base mesh; using the 3D positions of the markers to determine motion of the vertices of the base mesh; and constructing a texture map from the n camera images for each of multiple frames in the video sequence.
Claim 2
Independent Claim
152 words
A method for capturing a 3D geometric model of an animated 3D object, and color and shading information of a surface of the animated 3D object, the method comprising: capturing a base mesh of the 3D object; capturing n simultaneous video sequences of the animated 3D object with markers applied to the surface of the object from each of n cameras positioned around the 3D object, where n is two or more and the video sequences include n camera images per frame; determining 3D positions of the markers from the n 2D camera images for frames in the video sequence; determining an association between the 3D positions of the markers and vertices in the base mesh; using the 3D positions of the markers to determine motion of the vertices of the base mesh; and constructing a texture map from the n camera images for each of multiple frames in the video sequence.
Claim 3
Dependent Claim
85 words
The method of claim 1: identifying 2D locations of the markers in the n camera images for multiple frames in the video sequences by classifying colors of the markers in color classes, each color class comprising a list of colors associated with a marker color, evaluating color values of pixels in the camera images to determine whether to assign the pixels to a color class, and analyzing groups of adjacent pixels assigned to the same color class to determine whether the groups represent a marker.
Claim 4
Dependent Claim
89 words
The method of claim 1 including: a) identifying the 2D locations of the markers in the n camera images for a frame in the video sequences; b) determining the correspondence among the markers for the frame using the 2D locations of the markers in each of the n camera images for the frame; c) determining 3D positions for the markers for the frame using the correspondence among the markers and the 2D locations; d) repeating steps a) to c) to determine 3D positions for the markers for multiple frames.
Claim 5
Dependent Claim
50 words
The method of claim 3 further including: (e) determining correspondence among the 3D positions of the markers for a first frame and a second frame; and (f) repeating step (e) for additional pairs of frames such that each marker has a corresponding 3D position for each of the multiple frames.
Claim 6
Dependent Claim
29 words
The method of claim 1 wherein each of the texture maps has a corresponding set of 3D positions of the markers that define a deformation of the base mesh.
Claim 7
Dependent Claim
40 words
The method of claim 5 wherein each of the multiple frames has n 2D camera images, and the texture map and corresponding set of 3D positions for each frame are derived from the n 2D camera images for the frame.
Claim 8
Dependent Claim
97 words
The method of claim 5 wherein: each set of the 3D positions corresponds to one of the frames in the video sequence, each of the sets of 3D positions is used to deform the base mesh to a position of the 3D object at the corresponding frame, and the deformed mesh is used to determine how texels in a texture space for the corresponding frame are transformed to locations in a camera image to determine color values for the texels based on color values of pixels in the camera image located at or near the transformed texels.
Claim 9
Dependent Claim
44 words
The method of claim 1 wherein the step of constructing a texture map includes: computing a preliminary texture map from each of the n camera images; and computing the texture map for the frame as a weighted average of the n preliminary texture maps.
Claim 10
Dependent Claim
78 words
The method of claim 8 wherein each of the multiple frames has a corresponding set of 3D positions of the markers, and further including: using the set of 3D positions for the markers to determine a deformation of the base mesh for the corresponding frame; and using the deformation to determine a mapping of texels in each of the preliminary texture maps to the respective camera images to determine color values for the texels from the camera images.
Claim 11
Dependent Claim
35 words
The method of claim 1 further including: replacing portions of the camera images depicting reference markers with a texture created from an image of a portion of the 3D object not covered by reference marker.
Claim 12
Dependent Claim
57 words
The method of claim 1 further including: representing the motion of the base mesh as sets of deformation vectors, each set of deformation vectors corresponding to a position of the base mesh; decomposing a matrix of the sets of deformation vectors into basis vectors and coefficients; and coding the basis vectors and coefficients for transmission or storage.
Claim 13
Dependent Claim
13 words
The method of claim 11 further including: performing temporal prediction on the coefficients.
Claim 14
Dependent Claim
14 words
The method of claim 11 further including: quantizing the basis vectors and the coefficients.
Claim 15
Dependent Claim
15 words
The method of claim 11 further including: entropy coding the basis vectors and the coefficients.
Claim 16
Independent Claim
118 words
A method for capturing a 3D geometric motion model of an animated 3D object from video sequences of the animated 3D object with markers applied to the 3D object, taken from n cameras positioned around the 3D object, where n is two or more; the method comprising: determining 3D positions of the markers from n 2D camera images for frames in the video sequence; determining an association between the 3D positions of the markers and positions in a base 3D model of the object; using the 3D positions of the markers to determine motion of the base 3D model; and constructing a texture map from the n camera images for each of multiple frames in the video sequences.
Claim 17
Independent Claim
118 words
A method for capturing a 3D geometric motion model of an animated 3D object from video sequences of the animated 3D object with markers applied to the 3D object, taken from n cameras positioned around the 3D object, where n is two or more; the method comprising: determining 3D positions of the markers from n 2D camera images for frames in the video sequence; determining an association between the 3D positions of the markers and positions in a base 3D model of the object; using the 3D positions of the markers to determine motion of the base 3D model; and constructing a texture map from the n camera images for each of multiple frames in the video sequences.
Claim 18
Dependent Claim
23 words
The method of claim 15 wherein each of the texture maps is associated with a corresponding set of 3D positions of the markers.
Claim 19
Dependent Claim
39 words
The method of claim 16 wherein each texture map and corresponding set of 3D positions of the markers are derived from an associated frame of the video sequences, where each frame of the video sequence includes n camera images.
Claim 20
Dependent Claim
74 words
The method of claim 17 wherein the texture maps are created by: deforming the base model according to the set of 3D positions of the associated frame; determining a mapping between a location in a camera image and a texel location in texture space using the deformed base model; and assigning a color value to the texel based on a color value of a pixel at or near the location in the camera image.
Claim 21
Dependent Claim
59 words
The method of claim 18, further including: repeating the steps of determining a mapping and assigning a color value for the texels in preliminary texture maps, where each preliminary texture map is derived from a corresponding camera image for the frame; and computing a weighted average of the preliminary texture maps to determine a texture map for the frame.
Claim 22
Dependent Claim
62 words
The method of claim 15 further including: representing the motion of the base model as sets of 3D positions associated with the base model, each set of 3D positions corresponding to a position of the base model; decomposing a matrix of the sets of 3D positions into basis vectors and coefficients; and coding the basis vectors and coefficients for transmission or storage.
Claim 23
Independent Claim
161 words
A method for generating animation of facial expressions comprising: retrieving sets of 3D positions of 3D reference points associated with a base 3D model of a face, where the sets of the 3D positions are derived from n camera images of n video sequences simultaneously captured of the face with reference markers applied to the face; retrieving texture maps, where each texture map is associated with a set of 3D positions and the texture maps are derived from frames of the video sequences; determining a position of the base 3D model of the face for an output image by deforming the base model according to a first set of 3D positions; rendering the deformed base model to the output image, including texture mapping a texture map associated with the first set of 3D positions onto the deformed base model; and repeating the rendering step for successive sets of 3D positions to produce a sequence of output images that simulate facial expression.
Claim 24
Independent Claim
161 words
A method for generating animation of facial expressions comprising: retrieving sets of 3D positions of 3D reference points associated with a base 3D model of a face, where the sets of the 3D positions are derived from n camera images of n video sequences simultaneously captured of the face with reference markers applied to the face; retrieving texture maps, where each texture map is associated with a set of 3D positions and the texture maps are derived from frames of the video sequences; determining a position of the base 3D model of the face for an output image by deforming the base model according to a first set of 3D positions; rendering the deformed base model to the output image, including texture mapping a texture map associated with the first set of 3D positions onto the deformed base model; and repeating the rendering step for successive sets of 3D positions to produce a sequence of output images that simulate facial expression.
Claim 25
Dependent Claim
38 words
The method of claim 21 wherein each set of 3D positions has a corresponding texture map, and each respective set of 3D positions and corresponding texture map are derived from a single, common frame in the video sequence.
Claim 26
Dependent Claim
24 words
The method of claim 22 each set of 3D positions and corresponding texture map are derived from n camera images for the common frame.
Claim 27
Dependent Claim
25 words
The method of claim 21 wherein the sets of 3D positions are derived from 2D locations of the reference markers in the n camera images.
Claim 28
Dependent Claim
23 words
The method of claim 21 further including: decompressing coded texture maps using a video decoder to reconstruct a series of the texture maps.
Claim 29
Dependent Claim
22 words
The method of claim 21 further including: reconstructing a matrix of the 3D positions from a matrix of basis vectors and coefficients.
Claim 30
Dependent Claim
20 words
The method of claim 26 further including: reconstructing basis vectors from corresponding samples of quantized and entropy coded basis vectors.
Claim 31
Dependent Claim
13 words
The method of claim 26 further including: reconstructing coefficients from temporally predicted coefficients.
References Cited (Backward Citations)
US5524187A
Cited
US5675720A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-06-06 | 2003-12-06 | 2004-06-06 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-06-06 | 2007-12-06 | 2008-06-06 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-06-06 | 2011-12-06 | 2012-06-06 | $7700.00 | Window Closed / Lapsed |