US6950104B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Methods and systems for animating facial features, and methods and systems for expression transformation
Assignee: Zhigu Holdings Ltd
•
Filing: 2000-08-30
•
Granted: 2005-09-27
•
Expires: 2020-08-30
Abstract
Abstract Methods and systems for animating facial features and transforming facial expressions are described. In one embodiment, a code book contains data that defines a set of facial expressions of a first person. A training set of facial expressions from a second person and corresponding expressions from the code book are used to derive a transformation function that is then applied to all of the expressions of the code book. In this manner, expressions from the first person can be realistically transformed into expressions of a second person and vice versa. Particularly advantageous aspects of the described embodiments provide a single common generic face model that is used as the basis for a fitting operation for many different faces. Use of the single common generic face model and certain user-defined constraints provide a mechanism by which correspondences between the different faces can be established. These correspondences provide a basis for facial animation operations, among which are included expression transformation.
Inventors
- 👤 Stephen Marschner
- 👤 Brian K. Guenter
- 👤 Sashi Raghupathy
- 👤 Kirk Olynyk
- 👤 Sing Bing Kang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 13:33
CPC Classifications
G; G06; G06T; G06T13/00; G06T13/20; G06T13/40; G06T15/00; G06T15/04; G06T17/00; G06T17/10; G06T17/20; G06T17/205; G06T2200/00; G06T2200/08; Y; Y10; Y10S; Y10S345/00; Y10S345/949; Y10S345/956
Claims Analysis
Total Claims: 55
Claim 1
Independent Claim
92 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 2
Dependent Claim
92 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 3
Dependent Claim
19 words
The method of claim 1 , wherein the training set of expressions contains fewer expressions than the code book.
Claim 4
Dependent Claim
27 words
The method of claim 1 , wherein the transformation function compensates for differences in the size and shape of the faces of the first and second persons.
Claim 5
Dependent Claim
25 words
The method of claim 1 , wherein said deriving of the transformation function comprises computing a linear transformation from one set of expressions to another.
Claim 6
Dependent Claim
110 words
The method of claim 1 , wherein the deriving of the transformation function comprises: representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g a1 . . . n, and a corresponding set of vectors for a target face, g b1 . . . n, by solving 3 m linear least squares systems of the following form: α j ·g ai =g bi [j ], i=1 . . . n.
Claim 7
Dependent Claim
47 words
The method of claim 1 , wherein said providing data is defining a second set of facial expressions comprises: illuminating the second person's face with illumination; and contemporaneously capturing structure data describing the face's structure and reflectance data describing reflectance properties of the face from the illumination.
Claim 8
Dependent Claim
74 words
One or more computer-readable media having computer-readable instructions thereon which, when executed by a computer, cause the computer to: operate on a training set of expressions from one person and corresponding expressions from a code book of another person to compute a linear transformation function from the training set and their corresponding expressions; and apply the transformation function to a plurality of expressions from the code book to provide a synthetic set of expressions.
Claim 9
Dependent Claim
74 words
One or more computer-readable media having computer-readable instructions thereon which, when executed by a computer, cause the computer to: operate on a training set of expressions from one person and corresponding expressions from a code book of another person to compute a linear transformation function from the training set and their corresponding expressions; and apply the transformation function to a plurality of expressions from the code book to provide a synthetic set of expressions.
Claim 10
Dependent Claim
33 words
The computer-readable media of claim 7 , wherein the instructions cause the computer to use the synthetic set of expressions to transform expressions from the one person into expressions of the other person.
Claim 11
Dependent Claim
31 words
The computer-readable media of claim 8 , wherein the instructions cause the computer to transform expressions from the one person that are different from those expressions comprising the code book expressions.
Claim 12
Dependent Claim
34 words
The computer-readable media of claim 8 , wherein the instructions cause the computer to transform expressions by transmitting at least one index of a synthetic expression to a receiver that can reconstruct the expression.
Claim 13
Dependent Claim
17 words
The computer-readable media of claim 8 , wherein the instructions cause the computer to transform facial expressions.
Claim 14
Dependent Claim
93 words
A facial expression transformation system comprising: a code book embodied on a computer-readable medium, the code book containing data defining a first set of facial expressions of a first person; data embodied on a computer-readable medium, the data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; and a transformation processor configured to derive a transformation function from the training set of expressions and corresponding expressions from the first set of expressions.
Claim 15
Dependent Claim
93 words
A facial expression transformation system comprising: a code book embodied on a computer-readable medium, the code book containing data defining a first set of facial expressions of a first person; data embodied on a computer-readable medium, the data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; and a transformation processor configured to derive a transformation function from the training set of expressions and corresponding expressions from the first set of expressions.
Claim 16
Dependent Claim
17 words
The expression transformation system of claim 12 , wherein the transformation processor comprises a linear transformation processor.
Claim 17
Dependent Claim
36 words
The expression transformation system of claim 12 further comprising a synthetic set of expressions embodied on a computer-readable medium, the synthetic set of expressions being derived by applying the transformation function to the code book expressions.
Claim 18
Dependent Claim
29 words
The expression transformation system of claim 12 , wherein the transformation function compensates for differences in the size and shape of the face of the first and second persons.
Claim 19
Dependent Claim
113 words
The expression transformation system of claim 12 , wherein the transformation processor derives the transformation function by: representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g aJ . . . n, and a corresponding set of vectors for a target face, g bI . . . n, by solving 3 m linear least squares systems of the following form: α j ·g aJ =g bi [j ], i=1 . . . n.
Claim 20
Dependent Claim
186 words
A facial expression transformation system comprising: a transmitter comprising: a facial illumination system that is configured to provide multiple different light sources at the same time for illuminating a subject's face; a data-capturing system configured to capture both structure data and reflectance data from the subject's face when illuminated by the facial illumination system; and a first code book of synthetic expressions that have been synthesized by: receiving a training set of expressions provided by the subject; computing a transformation function using the training set of expressions and corresponding unsynthesized code book expressions; and applying the transformation fiction to all of the expressions in the code book; and a receiver communicatively linked with the transmitter and comprising: a reconstruction module for reconstructing facial images; and a second code book containing the same synthetic expressions as the first code book; and the transmitter being configured to: capture additional expressions of the subject; search the first code book for a corresponding or near matching expression; and transmit an index of a corresponding or matching code book expression to the receiver for facial image reconstruction by the reconstruction module.
Claim 21
Dependent Claim
186 words
A facial expression transformation system comprising: a transmitter comprising: a facial illumination system that is configured to provide multiple different light sources at the same time for illuminating a subject's face; a data-capturing system configured to capture both structure data and reflectance data from the subject's face when illuminated by the facial illumination system; and a first code book of synthetic expressions that have been synthesized by: receiving a training set of expressions provided by the subject; computing a transformation function using the training set of expressions and corresponding unsynthesized code book expressions; and applying the transformation fiction to all of the expressions in the code book; and a receiver communicatively linked with the transmitter and comprising: a reconstruction module for reconstructing facial images; and a second code book containing the same synthetic expressions as the first code book; and the transmitter being configured to: capture additional expressions of the subject; search the first code book for a corresponding or near matching expression; and transmit an index of a corresponding or matching code book expression to the receiver for facial image reconstruction by the reconstruction module.
Claim 22
Dependent Claim
19 words
The expression transformation system of claim 17 , wherein the illumination system comprises at least one polarized light source.
Claim 23
Dependent Claim
17 words
The expression transformation system of claim 17 , wherein the illumination system comprises multiple polarized light sources.
Claim 24
Dependent Claim
26 words
The expression transformation system of claim 17 , wherein the illumination system comprises a patterned light source configured to project a pattern onto the subject's face.
Claim 25
Dependent Claim
27 words
The expression transformation system of claim 17 , wherein the illumination system comprises an infrared patterned light source configured to project a pattern onto the subject's face.
Claim 26
Dependent Claim
18 words
The expression transformation system of claim 17 , wherein the different light sources are all infrared light sources.
Claim 27
Independent Claim
33 words
A method of animating facial features comprising: defining a subdivision surface that approximates geometry of a plurality of different faces; and fitting the same subdivision surface to each of the plurality of faces.
Claim 28
Independent Claim
33 words
A method of animating facial features comprising: defining a subdivision surface that approximates geometry of a plurality of different faces; and fitting the same subdivision surface to each of the plurality of faces.
Claim 29
Dependent Claim
19 words
The method of claim 23 , wherein said defining comprises defining the subdivision surface with a coarse mesh structure.
Claim 30
Dependent Claim
15 words
The method of claim 24 , wherein the coarse mesh structure comprises a triangular mesh.
Claim 31
Dependent Claim
22 words
The method of claim 23 , wherein said fitting comprises performing a continuous optimization operation over vertex positions of the subdivision surface.
Claim 32
Dependent Claim
29 words
The method of claim 23 , wherein said fitting comprises fitting the subdivision surface to the faces without altering the connectivity of a; mesh that defines the subdivision surface.
Claim 33
Dependent Claim
22 words
The method of claim 23 wherein said fitting comprises minimizing a smoothing functional associated with a mesh that defines the subdivision surface.
Claim 34
Dependent Claim
35 words
The method of claim 23 , wherein said fitting comprises selecting one or more constraints associated with a mesh that defines the subdivision surface and fitting those constraints directly to corresponding points on the faces.
Claim 35
Dependent Claim
19 words
The method of claim 29 , wherein the constraints are associated with one of the eyes, nose and mouth.
Claim 36
Dependent Claim
21 words
The method of claim 23 , wherein said fitting comprises minimizing a functional that includes terms for distance, smoothness, and constraints.
Claim 37
Dependent Claim
17 words
The method of claim 23 , wherein said fitting comprises solving a sequence of linear least-squares problems.
Claim 38
Dependent Claim
31 words
One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more computers, cause the one or more computers to implement the method of claim 23 .
Claim 39
Dependent Claim
212 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions, wherein the deriving of the transformation function comprises: representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g a1 . . . n, and a corresponding set of vectors for a target face, g b1 . . . n, by solving 3 m linear least squares systems of the following form: α j ·g ai =g bi [j ], 8 =1 . . . n, wherein said computing comprises using only a subset of points for each g aj : and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 40
Dependent Claim
212 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions, wherein the deriving of the transformation function comprises: representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g a1 . . . n, and a corresponding set of vectors for a target face, g b1 . . . n, by solving 3 m linear least squares systems of the following form: α j ·g ai =g bi [j ], 8 =1 . . . n, wherein said computing comprises using only a subset of points for each g aj : and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 41
Dependent Claim
23 words
The method of claim 34 , wherein said using comprises using only points that share edges with a standard sample point under consideration.
Claim 42
Dependent Claim
213 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions, wherein the deriving of the transformation function comprises; representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g a1 . . . n, and a corresponding set of vectors for a target face, g b1 . . . n, by solving 3 m linear least squares systems of the following form: α j ·g ai =g bi [j ], i=1 . . . n; controlling the spread of singular values when computing a pseudoinverse to solve for the α j ; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 43
Dependent Claim
213 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions, wherein the deriving of the transformation function comprises; representing each expression as a 3 m-vector that contains x, y, z displacements at m standard sample positions; and computing a set of linear predictors α j , one for each coordinate of g a , given a set of n expression vectors for a face to be transformed, g a1 . . . n, and a corresponding set of vectors for a target face, g b1 . . . n, by solving 3 m linear least squares systems of the following form: α j ·g ai =g bi [j ], i=1 . . . n; controlling the spread of singular values when computing a pseudoinverse to solve for the α j ; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 44
Dependent Claim
33 words
The method of claim 36 , wherein said controlling the spread comprises zeroing out all singular values less than ασ 1 , where σ 1 is the largest singular value of the matrix.
Claim 45
Dependent Claim
200 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with illumination, said illuminating comprising: using multiple light sources, one of which projecting a pattern on the second person's face from which the structure data can be ascertained; at least one of the light sources comprising an infrared light source; at least one of the light sources being polarized; and contemporaneously capturing structure data describing the face's structure and reflectance data describing reflectance properties of the face from the illumination, said capturing comprising using a camera having a polarizer that suppresses specularly-reflected light so that diffuse component reflection data is captured; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide B synthetic set of expressions.
Claim 46
Dependent Claim
200 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with illumination, said illuminating comprising: using multiple light sources, one of which projecting a pattern on the second person's face from which the structure data can be ascertained; at least one of the light sources comprising an infrared light source; at least one of the light sources being polarized; and contemporaneously capturing structure data describing the face's structure and reflectance data describing reflectance properties of the face from the illumination, said capturing comprising using a camera having a polarizer that suppresses specularly-reflected light so that diffuse component reflection data is captured; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide B synthetic set of expressions.
Claim 47
Dependent Claim
171 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with a first polarized light source that is selected so that specularly-suppressed reflective properties of the face can be ascertained; illuminating the second person's face with a second structured light source that projects a pattern onto the face, while simultaneously illuminating the face with the first polarized light source; and capturing both specularly-suppressed reflection data and structure data from the simultaneous illumination; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 48
Dependent Claim
171 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with a first polarized light source that is selected so that specularly-suppressed reflective properties of the face can be ascertained; illuminating the second person's face with a second structured light source that projects a pattern onto the face, while simultaneously illuminating the face with the first polarized light source; and capturing both specularly-suppressed reflection data and structure data from the simultaneous illumination; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 49
Dependent Claim
15 words
The method of claim 39 , wherein the light sources provide light at different frequencies.
Claim 50
Dependent Claim
13 words
The method of claim 39 , wherein the light sources provide infrared light.
Claim 51
Dependent Claim
35 words
The method of claim 39 , further comprising processing the captured data to provide both (a) data that describes dimensional aspects of the face and (b) data that describes diffuse reflective properties of the face.
Claim 52
Dependent Claim
158 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with multiple different light sources; measuring range map data from said illuminating; measuring image data from said illuminating; deriving a 3-dimensional surface from the range map data; computing surface normals to the 3 dimensional surface; and processing the surface normals and the image data to derive an albedo map; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 53
Dependent Claim
158 words
A facial expression transformation method comprising: defining a code book containing data defining a first set of facial expressions of a first person; providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person, wherein said providing data defining a second set of facial expressions comprises: illuminating the second person's face with multiple different light sources; measuring range map data from said illuminating; measuring image data from said illuminating; deriving a 3-dimensional surface from the range map data; computing surface normals to the 3 dimensional surface; and processing the surface normals and the image data to derive an albedo map; deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and applying the transformation function to the first set of expressions to provide a synthetic set of expressions.
Claim 54
Dependent Claim
16 words
The method of claim 43 , wherein at least one of the light sources is polarized.
Claim 55
Dependent Claim
14 words
The method of claim 43 , wherein all of the light sources ar polarized.
References Cited (Backward Citations)
US6163322A
Cited
US6351269B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-09-26 | 2009-03-28 | 2009-09-27 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-09-26 | 2013-03-28 | 2013-09-27 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-09-26 | 2017-03-28 | 2017-09-27 | $7700.00 | Window Closed / Lapsed |