Patent US5774591A successfully refreshed from Google Patents.
US5774591A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images
Assignee: University of Maryland, Baltimore
•
Filing: 1995-12-15
•
Granted: 1998-06-30
•
Expires: 2015-12-15
Abstract
Abstract A system tracks human head and facial features over time by analyzing a sequence of images. The system provides descriptions of motion of both head and facial features between two image frames. These descriptions of motion are further analyzed by the system to recognize facial movement and expression. The system analyzes motion between two images using parameterized models of image motion. Initially, a first image in a sequence of images is segmented into a face region and a plurality of facial feature regions. A planar model is used to recover motion parameters that estimate motion between the segmented face region in the first image and a second image in the sequence of images. The second image is warped or shifted back towards the first image using the estimated motion parameters of the planar model, in order to model the facial features relative to the first image. An affine model and an affine model with curvature are used to recover motion parameters that estimate the image motion between the segmented facial feature regions and the warped second image. The recovered motion parameters of the facial feature regions represent the relative motions of the facial features between the first image and the warped image. The face region in the second image is tracked using the recovered motion parameters of the face region. The facial feature regions in the second image are tracked using both the recovered motion parameters for the face region and the motion parameters for the facial feature regions. The parameters describing the motion of the face and facial features are filtered to derive mid-level predicates that define facial gestures occurring between the two images. These mid-level predicates are evaluated over time to determine facial expression and gestures occurring in the image sequence.
Inventors
- 👤 Michael J. Black
- 👤 Yaser Yacoob
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 20:25
CPC Classifications
G; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161; G06V40/165; G06V40/174; G06V40/176
Claims Analysis
Total Claims: 35
Claim 1
Independent Claim
253 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for recognizing facial expression of the human face, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; and associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression formed during a first time period in the sequence of images.
Claim 2
Dependent Claim
253 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for recognizing facial expression of the human face, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; and associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression formed during a first time period in the sequence of images.
Claim 3
Dependent Claim
73 words
A method according to claim 1, further comprising the step of identifying a third series of image pairs in the sequence of image pairs with motion parameter values having a third temporal relationship, the third temporal relationship defining a first peak of a facial expression, the third series of image pairs being positioned between the first series of image pairs and the second series of image pairs in the sequence of image pairs.
Claim 4
Dependent Claim
26 words
A method according to claim 2, further comprising the step of identifying the first series of image pairs over four images in the sequence of images.
Claim 5
Dependent Claim
26 words
A method according to claim 3, further comprising the step of identifying the second series of image pairs over four images in the sequence of images.
Claim 6
Dependent Claim
25 words
A method according to claim 4, further comprising the step of identifying the third series of image pairs over six images in the image sequence.
Claim 7
Dependent Claim
197 words
A method according to claim 2, further comprising the steps of: identifying a fourth series of image pairs in the sequence of image pairs with motion parameter values having a fourth temporal relationship, the fourth temporal relationship defining a second onset of a facial expression; identifying a fifth series of image pairs in the sequence of image pairs with motion parameter values having a fifth temporal relationship, the fifth temporal relationship defining a second ending of a facial expression, the fifth series of image pairs being positioned subsequent to the fourth series of image pairs in the sequence of image pairs; identifying a sixth series of image pairs in the sequence of image pairs with motion parameter values having a sixth temporal relationship, the sixth temporal relationship defining a second peak of a facial expression, the sixth series of image pairs being positioned between the fourth series of image pairs and the fifth series of image pairs in the sequence of image pairs; and associating the second onset of a facial expression with the second ending of a facial expression to identify a second facial expression formed during a second time period in the sequence of images.
Claim 8
Dependent Claim
66 words
A method according to claim 6, further comprising the steps of: detecting an overlap between the first time period and the second time period, the overlap providing an indication of a conflict between the first facial expression and the second facial expression; and determining a dominant facial expression in response to an indication of a conflict between the first time period and the second time period.
Claim 9
Dependent Claim
47 words
A method according to claim 7, wherein said step of determining a dominant facial expression determines how frequently facial gestures defining the first facial expression repeat over the first time period and how frequently facial gestures defining the second facial expression repeat over the second time period.
Claim 10
Dependent Claim
39 words
A method according to claim 8, wherein said step of detecting an overlap between the first and the second time periods detects an eighty percent overlap of image pairs between the first time period and the second time period.
Claim 11
Dependent Claim
36 words
A method according to claim 8, wherein said step of detecting an overlap between the first and the second time periods detects the first time period within fifteen image pairs of detecting the second time period.
Claim 12
Dependent Claim
52 words
A method according to claim 1, further comprising the step of identifying in a leading image of an image pair in the sequence of images a location of a first region in which the human face is located and a second region in which a facial feature of the face is located.
Claim 13
Dependent Claim
26 words
A method according to claim 11, wherein said identifying step segments the first region and the second region in the leading image of an image pair.
Claim 14
Dependent Claim
179 words
A method according to claim 11, wherein said determining step comprises the steps of: computing a first transformation for each image pair for estimating a motion of the face between a first region in a leading image and a first region in the trailing image of each image pair; forming a warped image for each image pair by warping the trailing image of each image pair with the first transformation, said forming step registering a motion of the facial feature between the leading image and the warped image of each image pair relative to a motion of the face between the leading image and the trailing image of each image pair; computing a second transformation to estimate a motion of the facial feature between a second region in the leading image and a region in the warped image of each image pair; and combining the first transformation and the second transformation to estimate a motion of the facial feature between the second region in the leading image and a second region in the trailing image of each image pair.
Claim 15
Dependent Claim
35 words
A method according to claim 13, further comprising the step of tracking the first region and the second region between the leading image and the trailing image with the first transformation and the second transformation.
Claim 16
Dependent Claim
29 words
A method according to claim 14, further comprising the step of thresholding each motion parameter in the set of motion parameters to determine a set of predefined facial gesture.
Claim 17
Dependent Claim
24 words
A method according to claim 15, wherein said combining step evaluates the set of predefined facial gestures for temporal relationships corresponding to facial expression.
Claim 18
Dependent Claim
51 words
A method according to claim 16, wherein said step of identifying the first series of image pairs in the sequence of images comprises the steps of: detecting a first facial gesture triggering a unique onset of facial expression; and detecting a second facial gesture supporting the unique onset of facial expression.
Claim 19
Dependent Claim
25 words
A method according to claim 1, further comprising the step of providing a sequence of images having a sampling rate of thirty frames per second.
Claim 20
Independent Claim
273 words
A system for recognizing facial expression of the human face, the system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, comprising: a motion estimation system for defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs, said motion estimation system determining motion parameter values for each of the motion parameters of the parametric flow model defined for each image pair in the sequence of image pairs; and a first detector for detecting a first facial expression, said first detector including: means for identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; means for identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; and means for associating the first onset of facial expression with the first ending of a facial expression to identify the first facial expression formed during a first time period in the sequence of images.
Claim 21
Independent Claim
273 words
A system for recognizing facial expression of the human face, the system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, comprising: a motion estimation system for defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs, said motion estimation system determining motion parameter values for each of the motion parameters of the parametric flow model defined for each image pair in the sequence of image pairs; and a first detector for detecting a first facial expression, said first detector including: means for identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; means for identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; and means for associating the first onset of facial expression with the first ending of a facial expression to identify the first facial expression formed during a first time period in the sequence of images.
Claim 22
Dependent Claim
55 words
A system according to claim 19, further comprising an image segmentation system for identifying in a leading image of an image pair in the sequence of images a location of a first region in which the human face is located and a second region in which a facial feature of the human face is located.
Claim 23
Dependent Claim
30 words
A system according to claim 20, wherein said image segmentation system comprises means for segmenting the first region and the second region in the leading image of an image pair.
Claim 24
Dependent Claim
185 words
A system according to claim 20, wherein said motion estimation system comprises: means for computing a first transformation for each image pair for estimating a motion of the face between a first region in a leading image and a first region in the trailing image of each image pair; means for forming a warped image for each image pair by warping the trailing image of each image pair with the first transformation, said forming means registering a motion of the facial feature between the leading image and the warped image of each image pair relative to a motion of the face between the leading image and the trailing image of each image pair; means for computing a second transformation to estimate a motion of the facial feature between a second region in the leading image and a region in the warped image of each image pair; and means for combining the first transformation and the second transformation to estimate a motion of the facial feature between the second region in the leading image and a second region in the trailing image of each image pair.
Claim 25
Dependent Claim
34 words
A system according to claim 22, further comprising means for tracking the first region and the second region between the leading image and the trailing image with the first transformation and the second transformation.
Claim 26
Dependent Claim
32 words
A system according to claim 19, further comprising: a second detector for detecting a second facial expression; and an arbiter for resolving conflicting expressions detected between said first and said second detectors.
Claim 27
Dependent Claim
30 words
A system according to claim 24, wherein said arbiter comprises means for detecting a series of image pairs that define both the first facial expression and the second facial expression.
Claim 28
Dependent Claim
235 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for recognizing facial gestures, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction; identifying a second series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values, the second cyclical change in motion parameter values indicating a movement in a second direction; and associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identity a first facial gesture formed during a first time period in the sequence of images.
Claim 29
Dependent Claim
235 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for recognizing facial gestures, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by said defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction; identifying a second series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values, the second cyclical change in motion parameter values indicating a movement in a second direction; and associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identity a first facial gesture formed during a first time period in the sequence of images.
Claim 30
Dependent Claim
121 words
A method according to claim 26, further comprising the steps of: identifying a third series of image pairs in the sequence of image pairs with a third cyclical change in motion parameter values, the third cyclical change in motion parameter values indicating a movement in a first direction; identifying a fourth series of image pairs in the sequence of image pairs with a fourth cyclical change in motion parameter values, the fourth cyclical change in motion parameter values indicating a movement in a second direction; and associating the third cyclical change in motion parameter values with the fourth cyclical change in motion parameter values to identify a second facial gesture formed during a second time period in the sequence of images.
Claim 31
Dependent Claim
64 words
A method according to claim 27, further comprising the steps of: detecting an overlap between the first time period and the second time period, the overlap providing an indication of a conflict between the first gesture and the second gesture; and determining a dominant facial gesture in response to an indication of a conflict between the first time period and the second time period.
Claim 32
Dependent Claim
39 words
A method according to claim 28, wherein said step of detecting an overlap between the first and the second time periods detects an eighty percent overlap of image pairs between the first time period and the second time period.
Claim 33
Dependent Claim
36 words
A method according to claim 28, wherein said step of detecting an overlap between the first and the second time periods detects the first time period within fifteen image pairs of detecting the second time period.
Claim 34
Dependent Claim
340 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for simultaneously recognizing facial expression and facial gestures, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by slid defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; identifying a third series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction; identifying a fourth series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values the second cyclical change in motion parameter values indicating a movement in a second direction; associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identify a first facial gesture; and associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression.
Claim 35
Dependent Claim
340 words
In a system receiving a sequence of N images representing a human face moving over time, with the sequence of N images forming N-1 image pairs, each image pair in the sequence of N-1 image pairs having a leading image and a trailing image in the sequence of N images, a method for simultaneously recognizing facial expression and facial gestures, comprising the steps of: defining a parametric flow model with motion parameters for describing rigid and non-rigid movement of facial features between the leading image and the trailing image of each image pair in the sequence of image pairs; determining motion parameter values for each of the motion parameters of the parametric flow model defined by slid defining step for each image pair in the sequence of image pairs; identifying a first series of image pairs in the sequence of image pairs with motion parameter values having a first temporal relationship, the first temporal relationship defining a first onset of a facial expression; identifying a second series of image pairs in the sequence of image pairs with motion parameter values having a second temporal relationship, the second temporal relationship defining a first ending of a facial expression, the second series of image pairs being positioned subsequent to the first series of image pairs in the sequence of image pairs; identifying a third series of image pairs in the sequence of image pairs with a first cyclical change in motion parameter values, the first cyclical change in motion parameter values indicating a movement in a first direction; identifying a fourth series of image pairs in the sequence of image pairs with a second cyclical change in motion parameter values the second cyclical change in motion parameter values indicating a movement in a second direction; associating the first cyclical change in motion parameter values with the second cyclical change in motion parameter values to identify a first facial gesture; and associating the first onset of a facial expression with the first ending of a facial expression to identify a first facial expression.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2001-06-29 | 2001-12-29 | 2002-06-30 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2005-06-29 | 2005-12-29 | 2006-06-30 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2009-06-29 | 2009-12-29 | 2010-06-30 | $7700.00 | Window Closed / Lapsed |