US7755619B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Automatic 3D face-modeling from video
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2006-08-17
•
Granted: 2010-07-13
•
Expires: 2026-08-17
Abstract
Abstract Systems and methods perform automatic 3D face modeling. In one implementation, a brief video clip of a user's head turning from front to side provides enough input for automatically achieving a model that includes 2D feature matches, 3D head pose, 3D face shape, and facial textures. The video clip of the user may be of poor quality. In a two layer iterative method, the video clip is divided into segments. Flow-based feature estimation and model-based feature refinement are applied recursively to each segment. Then the feature estimation and refinement are iteratively applied across all the segments. The entire modeling method is automatic and the two layer iterative method provides speed and efficiency, especially when sparse bundle adjustment is applied to boost efficiency.
Inventors
- 👤 Qiang Wang
- 👤 Heung-Yeung Shum
- 👤 Xiaoou Tang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:24
CPC Classifications
G; G06; G06T; G06T17/00; G06T17/20; G06T7/00; G06T7/50; G06T7/55; G06T2200/00; G06T2200/08
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
89 words
A method, comprising: under control of one or more processors executing computer-executable instructions: receiving video information of a face rotating from a front view position to a side view position; creating a 3-dimension model of the face at least based in part on the video information, the creating including: dividing the video information into a plurality of segments; and calculating a length of each segment such that upon dividing the video information, the plurality of segments comprise at least one frame overlapping between successive segments for continued feature tracking.
Claim 2
Independent Claim
89 words
A method, comprising: under control of one or more processors executing computer-executable instructions: receiving video information of a face rotating from a front view position to a side view position; creating a 3-dimension model of the face at least based in part on the video information, the creating including: dividing the video information into a plurality of segments; and calculating a length of each segment such that upon dividing the video information, the plurality of segments comprise at least one frame overlapping between successive segments for continued feature tracking.
Claim 3
Dependent Claim
26 words
The method as recited in claim 1 , wherein receiving video information comprises capturing a video clip of the face under variable or poor lighting conditions.
Claim 4
Dependent Claim
28 words
The method as recited in claim 1 , wherein automatically creating a 3-dimension model of the face includes deriving values for parameters of a morphable 3D face model.
Claim 5
Dependent Claim
21 words
The method as recited in claim 1 , further comprising performing an automatic initialization to obtain a rough initial face pose.
Claim 6
Dependent Claim
123 words
The method as recited in claim 1 , wherein creating a 3-dimension model of the face further comprises: for each segment, to simultaneously estimate 2-dimensional feature matches, at least one 3-dimensional head pose, and at least one 3-dimensional face shape within each segment, iterating between: estimating features of the face across frames of the segment; refining the features across frames of the segment using the estimated features; iteratively optimizing the estimating features and the refining features according to a re-projection constraint; and iteratively combining feature correspondences across the segments to obtain 2-dimensional feature matches, at least one 3-dimensional head pose, at least one 3-dimensional face shape, and at least one facial texture for the face in all the segments of the video information.
Claim 7
Dependent Claim
27 words
The method as recited in claim 5 , wherein the estimating features overcomes errors in the video information due to noise-perturbation or poor illumination of the face.
Claim 8
Dependent Claim
29 words
The method as recited in claim 5 , wherein combining feature correspondences across segments approximately uses the estimating features and the refining features that is used for each segment.
Claim 9
Dependent Claim
17 words
The method as recited in claim 5 , wherein the estimating features comprises a flow-based model process.
Claim 10
Dependent Claim
51 words
The method as recited in claim 5 , wherein the estimating features comprises a flow-based model process that includes: iteratively estimating a head pose; iteratively estimating values for parameters of a morphable 3D face model; and iteratively performing 2-dimensional to 3-dimensional feature re-matching or iteratively performing 2-dimensional to 3-dimensional point re-matching.
Claim 11
Dependent Claim
17 words
The method as recited in claim 5 , wherein the refining features comprises a model-based flow process.
Claim 12
Dependent Claim
37 words
The method as recited in claim 5 , wherein the refining features comprises a model-based flow process that includes: iteratively image warping with estimated model parameters and estimated pose parameters; and iteratively performing small baseline feature matching.
Claim 13
Dependent Claim
25 words
The method as recited in claim 5 , wherein at least combining feature correspondences across the segments uses a sparse bundle adjustment to accelerate computation.
Claim 14
Dependent Claim
114 words
A 3-dimensional face-modeling engine, comprising: a segmenter to divide a video sequence of a face rotating from a frontal view to a side view into segments; a frame overlapper to calculate a length of each segment such that upon dividing the video sequence, the segments comprise at least one frame overlapping between successive segments for continued feature tracking; an iterator to iteratively estimate and iteratively refine facial features across images of each segment; and a post-processor to combine the facial features across the segments for the entire video sequence to obtain 2-dimensional feature matches, at least one 3-dimensional head pose, at least one 3-dimensional face shape, and at least one texture for the face.
Claim 15
Dependent Claim
114 words
A 3-dimensional face-modeling engine, comprising: a segmenter to divide a video sequence of a face rotating from a frontal view to a side view into segments; a frame overlapper to calculate a length of each segment such that upon dividing the video sequence, the segments comprise at least one frame overlapping between successive segments for continued feature tracking; an iterator to iteratively estimate and iteratively refine facial features across images of each segment; and a post-processor to combine the facial features across the segments for the entire video sequence to obtain 2-dimensional feature matches, at least one 3-dimensional head pose, at least one 3-dimensional face shape, and at least one texture for the face.
Claim 16
Dependent Claim
29 words
The 3-dimensional face-modeling engine as recited in claim 13 , wherein the iterator includes: a feature estimator comprising flow-based model components; and a feature refiner comprising model-based flow components.
Claim 17
Dependent Claim
29 words
The 3-dimensional face-modeling engine as recited in claim 14 , wherein the feature estimator includes: a pose estimator; a model parameter estimator; and a 2-dimensional to 3-dimensional feature re-matcher.
Claim 18
Dependent Claim
24 words
The 3-dimensional face-modeling engine as recited in claim 14 , wherein the feature refiner includes: an image-pair pre-warping engine; and a small-baseline features matcher.
Claim 19
Dependent Claim
38 words
The 3-dimensional face-modeling engine as recited in claim 13 , further comprising an initializer to detect a face in a first frame of at least one of the segments and to extract facial features of the detected face.
Claim 20
Dependent Claim
23 words
The 3-dimensional face-modeling engine as recited in claim 13 , wherein at least the post-processor uses a sparse bundle adjustment to accelerate computations.
Claim 21
Dependent Claim
21 words
The 3-dimensional face-modeling engine as recited in claim 13 , wherein the iterator uses an optimization that includes a re-projection constraint.
Claim 22
Independent Claim
87 words
A system, comprising: means for receiving video information of a face rotating from a frontal view to a side view; means for creating a 3-dimension model of the face based at least in part on the video information, the means for creating including: means for dividing the video information into a plurality of segments; and means for calculating a length of each segment such that upon dividing the video information, the plurality of segments comprise at least one frame overlapping between successive segments for continued feature tracking.
Claim 23
Independent Claim
87 words
A system, comprising: means for receiving video information of a face rotating from a frontal view to a side view; means for creating a 3-dimension model of the face based at least in part on the video information, the means for creating including: means for dividing the video information into a plurality of segments; and means for calculating a length of each segment such that upon dividing the video information, the plurality of segments comprise at least one frame overlapping between successive segments for continued feature tracking.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2013-07-12 | 2014-01-11 | 2014-07-13 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2017-07-12 | 2018-01-11 | 2018-07-13 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2021-07-12 | 2022-01-11 | 2022-07-13 | $7700.00 | Window Closed / Lapsed |