Patent US7236615B2 successfully refreshed from Google Patents.
US7236615B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Synergistic face detection and pose estimation with energy-based models
Assignee: NEC Corp
•
Filing: 2005-03-31
•
Granted: 2007-06-26
•
Expires: 2025-03-31
Abstract
Abstract A method for human face detection that detects faces independently of their particular poses and simultaneously estimates those poses. Our method exhibits an immunity to variations in skin color, eyeglasses, facial hair, lighting, scale and facial expressions, and others. In operation, we train a convolutional neural network to map face images to points on a face manifold, and non-face images to points far away from that manifold, wherein that manifold is parameterized by facial pose. Conceptually, we view a pose parameter as a latent variable, which may be inferred through an energy-minimization process. To train systems based upon our inventive method, we derive a new type of discriminative loss function that is tailored to such detection tasks. Our method enables a multi-view detector that can detect faces in a variety of poses, for example, looking left or right (yaw axis), up or down (pitch axis), or tilting left or right (roll axis). Systems employing our method are highly-reliable, run at near real time (5 frames per second on conventional hardware), and is robust against variations in yaw (±90°), roll(±45°), and pitch(±60°).
Inventors
- 👤 Matthew L. Miller
- 👤 Margarita Osadchy
- 👤 Yann LeCun
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 23:10
CPC Classifications
G; G06; G06V; G06V20/00; G06V20/60; G06V20/64; G06V20/653; G06V10/00; G06V10/40; G06V10/44; G06V10/443; G06V10/449; G06V10/451; G06V10/454; G06V40/00; G06V40/10; G06V40/16; G06V40/161; G06V40/165
Claims Analysis
Total Claims: 6
Claim 1
Independent Claim
212 words
A computer-implemented method of face detection and pose estimation, the method comprising the following steps: training a convolutional neural network to map facial images to points on a face manifold, parameterized by facial pose, and to map non-facial images to points away from the face manifold; and simultaneously determining, whether an image is a face from its proximity to the face manifold and an estimate of facial pose of that image from its projection to the face manifold; wherein the training step further comprises the step(s) of: optimizing a loss function of three variables, wherein said variables include image, pose, and face/non-face characteristics of an image; wherein the loss function is represented by: Loss ( W ) = 1 S 1 ∑ i εS 1 L 1 ( W , Z i , X i ) + 1 S 0 ∑ i εS 1 L 0 ( W , X i ) ; where S 1 is the set of training faces, S 0 is the set of non-faces, L 3 (W,Z 1 ,X 1 ) and L 0 (W,X 1 ) are loss functions for a face sample (with a known pose) and non-face sample, respectively.
Claim 2
Dependent Claim
212 words
A computer-implemented method of face detection and pose estimation, the method comprising the following steps: training a convolutional neural network to map facial images to points on a face manifold, parameterized by facial pose, and to map non-facial images to points away from the face manifold; and simultaneously determining, whether an image is a face from its proximity to the face manifold and an estimate of facial pose of that image from its projection to the face manifold; wherein the training step further comprises the step(s) of: optimizing a loss function of three variables, wherein said variables include image, pose, and face/non-face characteristics of an image; wherein the loss function is represented by: Loss ( W ) = 1 S 1 ∑ i εS 1 L 1 ( W , Z i , X i ) + 1 S 0 ∑ i εS 1 L 0 ( W , X i ) ; where S 1 is the set of training faces, S 0 is the set of non-faces, L 3 (W,Z 1 ,X 1 ) and L 0 (W,X 1 ) are loss functions for a face sample (with a known pose) and non-face sample, respectively.
Claim 3
Dependent Claim
68 words
The method of claim 1 , wherein said determination step comprises the step(s) of: clamping X to an observed value (the image), and finding the values of Z and Y that minimize an energy function E W (Y,Z,X) according to the following relationship: ( Y , Z )= arg min Yε{Y},Zε{Z} E W ( Y,Z,X ) where {Y}={0, 1} and {Z}=[−90, 90]×[−45, 45] for yaw and roll variables.
Claim 4
Dependent Claim
58 words
The method of claim 2 , wherein the energy function for a face E W (1,Z,X), is the distance between the point produced by the network G W (X) and the point with pose Z on the manifold F(Z) according to the following relationship: E W (1 ,Z,X )=∥ G W ( X )− F ( Z )∥.
Claim 5
Dependent Claim
52 words
The method of claim 3 wherein the energy function for a non-face E W (0,Z,X), is equal to a threshold constant T, independent of Z and X that may be represented by the following relationship: E W ( Y,Z,X )= Y∥G W ( X )− F ( Z )∥+(1 −Y ) T.
Claim 6
Dependent Claim
72 words
The method of claim 1 wherein the determination step comprises: finding an output label and pose having the smallest energy according to the following relationship: Z =arg min Zε{Z} ∥G W ( X )− F ( Z )∥+)1 −Y ) T; comparing this minimum distance, ∥G W (X)−F( Z )∥, to threshold T; classifying X as a face if the minimum distance is smaller than T, otherwise classifying X as a non-face.
References Cited (Backward Citations)
US5649068A
Cited
US6959109B2
Cited
US20040264745A1
Cited
US20050180626A1
Cited
US20060034495A1
Cited
US20060078172A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2010-06-25 | 2010-12-25 | 2011-06-26 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2014-06-25 | 2014-12-25 | 2015-06-26 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2018-06-25 | 2018-12-25 | 2019-06-26 | $7700.00 | Window Closed / Lapsed |