US6504546B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method of modeling objects to synthesize three-dimensional, photo-realistic animations
Assignee: Rakuten Group Inc
•
Filing: 2000-02-08
•
Granted: 2003-01-07
•
Expires: 2020-02-08
Abstract
Abstract A method for modeling three-dimensional objects to create photo-realistic animations using a data-driven approach. The three-dimensional object is defined by a set of separate three-dimensional planes, each plane enclosing an area of the object that undergoes visual changes during animation. Recorded video is used to create bitmap data to populate a database for each three-dimensional plane. The video is analyzed in terms of both rigid movements (changes in pose) and plastic deformation (changes in expression) to create the bitmaps. The modeling is particularly well-suited for animations of a human face, where an audio track generated by a text-to-speech synthesizer can be added to the animation to create a photo-realistic "talking head".
Inventors
- 👤 Eric Cosatto
- 👤 Hans Peter Graf
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:34
CPC Classifications
G; G06; G06T; G06T13/00; G06T13/20; G06T13/40; G10; G10L; G10L21/00; G10L21/06; G10L21/10; G10L2021/105
Claims Analysis
Total Claims: 11
Claim 1
Independent Claim
214 words
A method of creating a three-dimensional model of a three-dimensional object for generating photo-realistic animation, the method comprising the steps of: a) defining a set of three-dimensional planes approximating an overall shape of the three-dimensional object and its surrounding area; b) recording a set of images of said three-dimensional object as it undergoes both rigid movement, defined as changes in pose, and plastic deformation, defined as changes in appearances; c) creating a bitmap for each recorded image; d) storing the created bitmaps in a database, the database used to create the three-dimensional model of said three-dimensional object; e) preparing an empty frame to receive a synthesized view of said three-dimensional object; f) choosing an arbitrary pose for the synthesis of said three-dimensional object and a desired appearance; g) selecting, for a three-dimensional plane from the set of three-dimensional planes, a bitmap with the desired appearance, chosen in step f), from those stored in the database and determining the pose of said three-dimensional object as it appears on that bitmap; h) re-projecting the three-dimensional plane of said three-dimensional object from its pose on the recorded bitmap onto its new pose on the empty frame created in step e); and i) repeating steps g) and h) for each three-dimensional plane in the set of three-dimensional planes.
Claim 2
Independent Claim
214 words
A method of creating a three-dimensional model of a three-dimensional object for generating photo-realistic animation, the method comprising the steps of: a) defining a set of three-dimensional planes approximating an overall shape of the three-dimensional object and its surrounding area; b) recording a set of images of said three-dimensional object as it undergoes both rigid movement, defined as changes in pose, and plastic deformation, defined as changes in appearances; c) creating a bitmap for each recorded image; d) storing the created bitmaps in a database, the database used to create the three-dimensional model of said three-dimensional object; e) preparing an empty frame to receive a synthesized view of said three-dimensional object; f) choosing an arbitrary pose for the synthesis of said three-dimensional object and a desired appearance; g) selecting, for a three-dimensional plane from the set of three-dimensional planes, a bitmap with the desired appearance, chosen in step f), from those stored in the database and determining the pose of said three-dimensional object as it appears on that bitmap; h) re-projecting the three-dimensional plane of said three-dimensional object from its pose on the recorded bitmap onto its new pose on the empty frame created in step e); and i) repeating steps g) and h) for each three-dimensional plane in the set of three-dimensional planes.
Claim 3
Dependent Claim
27 words
The method as defined in claim 1 wherein the method further comprises the step of: j) repeating steps e) through i) for each frame of an animation.
Claim 4
Dependent Claim
106 words
The method as defined in claim 1 , wherein in performing step h), the following steps are performed: 1) obtaining, for a selected plane from the set of three-dimensional planes, a first quadrilateral boundary marking the plane's perspective projection on the recorded image, using the pose of the object on said image; 2) obtaining, for the selected plane, a second quadrilateral boundary on the synthesized frame, marking the plane's perspective projection using the new chosen pose of the object; and 3) warping the pixels of the image that are bound by the first quadrilateral into the pixels of the synthesized frame bound by the second quadrilateral.
Claim 5
Dependent Claim
22 words
The method as defined in claim 3 wherein in performing step h 3 ), bilinear interpolation is used in the warping process.
Claim 6
Dependent Claim
25 words
The method as defined in claim 3 wherein in performing step h 3 ), pixel warping includes the method of alpha-blending using a transparency mask.
Claim 7
Dependent Claim
21 words
The method as defined by claim 1 wherein in performing step g), a feature vector is used for selecting the bitmap.
Claim 8
Dependent Claim
102 words
The method as defined in claim 6 wherein the feature vector contains one or more of the following features: the pose of the object; measurements of recognizable parts of the object obtained by computer vision algorithms from the image; statistics calculated directly from the pixels of the image; normalized bitmaps from which pixels are obtained by warping the pixels of an area of the image corresponding to the perspective projection of a plane of the object's model using the pose of the object; measurements of recognizable parts on a normalized bitmap; and statistics calculated directly from the pixels of a normalized bitmap.
Claim 9
Dependent Claim
29 words
The method as defined in claim 7 wherein the feature vectors are pre-computed from the recorded bitmaps of step b) and stored in the database, before executing step g).
Claim 10
Dependent Claim
42 words
The method as defined in claim 1 wherein the object is a human face and in performing step a), the set of three-dimensional planes approximating the overall shape of the face comprises: forehead, eyes, cheeks, mouth, chin, jaw and the entire head.
Claim 11
Dependent Claim
35 words
The method as defined in claim 9 wherein in performing step f), the pose is computed from the positions of a set of facial parts, namely: the four eye corners and the pair of nostrils.
References Cited (Backward Citations)
US5933151A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-01-06 | 2006-07-08 | 2007-01-07 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-01-06 | 2010-07-08 | 2011-01-07 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-01-06 | 2014-07-08 | 2015-01-07 | $7700.00 | Window Closed / Lapsed |