US6014625A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for producing lip-movement parameters in a three-dimensional-lip-model
Assignee: Quarterhill Inc
•
Filing: 1997-12-02
•
Granted: 2000-01-11
•
Expires: 2017-12-02
Abstract
Abstract In a three-dimensional model-based coding system, a shape of mouth, especially, a shape of lip, among a shape of face, has an important visual information. As each person has the different shape of lip, however, it is desirable to transform the shape of lip based on an individual person. First, a characteristic three-dimensional lip model based on the individual person is generated by matching a three-dimensional lip model to a basic lip of the individual person. Subsequently, the characteristic three-dimensional lip model is converted into a two-dimensional lip model and rotated according an angle of a actual lip image. In case that the person is talking, the two-dimensional lip model is reconstructed based on a recognized syllable. Finally, a difference between the two-dimensional lip model and the actual lip image is calculated. As not a whole real lip image but the difference between the actual image and the two-dimensional lip model similar to the actual lip image is transmitted, the amount of transmission data can be reduced.
Inventors
- 👤 Min-Sup Lee
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 11:21
CPC Classifications
G; G06; G06T; G06T17/00
Claims Analysis
Total Claims: 22
Claim 1
Independent Claim
230 words
A method for producing a lip movement parameter by using voice signals and image signals in a 3 dimension model-based coding system, wherein the lip movement parameter is a distance between a actual lip image and a transformed 2 dimension lip model, the actual lip image being an image of a lip of a person displayed which varies continuously, the method comprising the steps of: (a) providing a characteristic 3 dimension lip model for the person by using a basic facial image of the person and a location of the basic lip of the person and a general 3 dimension lip model, wherein the basic facial image is an expressionless and mute front facial image of the person; (b) recognizing syllables from the voice signals of the person, and providing a first selection signal at a moment when an amount of an energy of the syllable along a time axis and a frequency axis becomes greater than a predetermined threshold, and providing a second selection signal at a moment when the amount of the energy of the syllable begins to fall below the predetermined threshold; (c) extracting the actual lip image of the person from the image signals; (d) presenting a 2 dimension lip model transformed based on a rotation angle and the syllable; and (e) transmitting lip movement parameters, the voice signals and the rotation angle to a decoder.
Claim 2
Independent Claim
230 words
A method for producing a lip movement parameter by using voice signals and image signals in a 3 dimension model-based coding system, wherein the lip movement parameter is a distance between a actual lip image and a transformed 2 dimension lip model, the actual lip image being an image of a lip of a person displayed which varies continuously, the method comprising the steps of: (a) providing a characteristic 3 dimension lip model for the person by using a basic facial image of the person and a location of the basic lip of the person and a general 3 dimension lip model, wherein the basic facial image is an expressionless and mute front facial image of the person; (b) recognizing syllables from the voice signals of the person, and providing a first selection signal at a moment when an amount of an energy of the syllable along a time axis and a frequency axis becomes greater than a predetermined threshold, and providing a second selection signal at a moment when the amount of the energy of the syllable begins to fall below the predetermined threshold; (c) extracting the actual lip image of the person from the image signals; (d) presenting a 2 dimension lip model transformed based on a rotation angle and the syllable; and (e) transmitting lip movement parameters, the voice signals and the rotation angle to a decoder.
Claim 3
Dependent Claim
131 words
The method according to claim 1, wherein the step (a) includes the steps of: (a1) selecting feature points on a contour of the 3 dimension lip model and assigning a high priority to them; (a2) moving the feature points onto a contour of the basic lip image by searching a most similar region between the 3 dimension lip model and the basic lip image within a regular search range; (a3) reconstructing the contour of the 3 dimension lip model on the basis of the moved feature points; and (a4) repeating the steps (a2) and (a3) with respect to remaining points in case that a distance between a point of the reconstructed 3 dimension lip model and its most similar region of the basic lip image is greater than a predetermined threshold.
Claim 4
Dependent Claim
60 words
The method according to claim 2, wherein the feature points include left and right ending points of the 3 dimension lip model, sharp-edged points in a upper region of the 3 dimension lip model and points which cross a central region of the 3 dimension lip model lengthwise among points located on the contour of the 3 dimension lip model.
Claim 5
Dependent Claim
17 words
The method according to claim 3, wherein the step (c) is carried out by a segmentation method.
Claim 6
Dependent Claim
79 words
The method according to claim 4, wherein the step (d) includes steps of: (d1) transforming the characteristic 3 dimension lip model into a characteristic 2 dimension lip model; (d2) calculating the rotation angle by comparing the characteristic 2 dimension lip model with the actual lip image to provide a rotated 2 dimension lip model and the rotation angle; and (d3) reconstructing the rotated 2 dimension lip model based on the syllable to provide a reconstructed 2 dimension lip model.
Claim 7
Dependent Claim
21 words
The method according to claim 5, wherein the reconstructed 2 dimension lip model is reconstructed based on 7 basic voice syllables.
Claim 8
Dependent Claim
18 words
The method according to claim 6, wherein the rotation angle is calculated by using an affine transform method.
Claim 9
Dependent Claim
88 words
The method according to claim 7, wherein the step (e) includes steps of: (e1) computing movement quantities within a predetermined computing range between the actual lip image and either one of the reconstructed 2 dimension lip model in response to the first selection signal provided from the step (b) and the rotated 2 dimension lip model in response to the second selection signal provided from the step (b) to generate the lip movement parameters; and (e2) multiplexing the lip movement parameters, the voice signals and the rotation angle.
Claim 10
Dependent Claim
56 words
The method according to claim 8, wherein the lip movement parameters include left-right movement parameters and up-down movement parameters of both left and right ending points of the 2 dimension lip model, and up-down movement parameters and forward-backward movement parameters of most upper and lower points in a central region of the 2 dimension lip model.
Claim 11
Dependent Claim
16 words
The method according to claim 9, wherein the predetermined computing range is between -1 and 1.
Claim 12
Independent Claim
232 words
An apparatus for producing a lip movement parameter by using voice signals and image signals in a 3 dimension model-based coding system, wherein the lip movement parameter is a distance between a actual lip image and a transformed 2 dimension lip model, the actual lip image being an image of a lip of a person displayed which varies continuously, comprising: means for recognizing syllables from the voice signals of the person to provide a recognized syllable and a selection signal; means for extracting the actual lip image of the person from the image signals; means for providing a characteristic 3 dimension lip model for the person by using a basic facial image of the person and a location of the basic lip of the person and a general 3 dimension lip model, wherein the basic facial image is an expressionless and mute front facial image of the person; means for transforming the characteristic 3 dimension lip model into a characteristic 2 dimension lip model, and calculating a rotation angle by comparing the characteristic 2 dimension lip model with the actual lip image to provide a rotated 2 dimension lip model and the rotation angle; means for reconstructing the rotated 2 dimension lip model based on the syllable to provide a reconstructed 2 dimension lip model; and means for transmitting lip movement parameters, the voice signals and the rotation angle to a decoder.
Claim 13
Independent Claim
232 words
An apparatus for producing a lip movement parameter by using voice signals and image signals in a 3 dimension model-based coding system, wherein the lip movement parameter is a distance between a actual lip image and a transformed 2 dimension lip model, the actual lip image being an image of a lip of a person displayed which varies continuously, comprising: means for recognizing syllables from the voice signals of the person to provide a recognized syllable and a selection signal; means for extracting the actual lip image of the person from the image signals; means for providing a characteristic 3 dimension lip model for the person by using a basic facial image of the person and a location of the basic lip of the person and a general 3 dimension lip model, wherein the basic facial image is an expressionless and mute front facial image of the person; means for transforming the characteristic 3 dimension lip model into a characteristic 2 dimension lip model, and calculating a rotation angle by comparing the characteristic 2 dimension lip model with the actual lip image to provide a rotated 2 dimension lip model and the rotation angle; means for reconstructing the rotated 2 dimension lip model based on the syllable to provide a reconstructed 2 dimension lip model; and means for transmitting lip movement parameters, the voice signals and the rotation angle to a decoder.
Claim 14
Dependent Claim
91 words
The apparatus according to claim 11, wherein the recognizing means includes: means for converting the voice signals into digitalized voice signals; means for identifying syllables from the digitalized voice signals; and means for presenting a first selection signal at a moment when an amount of an energy of the recognized syllable along a time axis and a frequency axis becomes greater than a predetermined threshold, and presenting a second selection signal at a moment when the amount of the energy of the recognized syllable begins to fall below the predetermined threshold.
Claim 15
Dependent Claim
14 words
The apparatus according to claim 12, wherein the extracting means includes a segmentation technique.
Claim 16
Dependent Claim
132 words
The apparatus according to claim 13, wherein the providing means includes: means for selecting feature points on a contour of the 3 dimension lip model and assigning a high priority to them; means for moving the feature points onto a contour of the basic lip image by searching a most similar region between the 3 dimension lip model and the basic lip image within a regular search range; means for reconstructing the contour of the 3 dimension lip model on the basis of the moved feature points; and means for repeating the moving and reconstructing means with respect to remaining points in case that a distance between a point of the reconstructed 3 dimension lip model and its most similar region of the basic lip image is greater than a predetermined threshold.
Claim 17
Dependent Claim
60 words
The apparatus according to claim 14, wherein the feature points include left and right ending points of the 3 dimension lip model, sharp-edged points in a upper region of the 3 dimension lip model and points which cross a central region of the 3 dimension lip model lengthwise among points located on the contour of the 3 dimension lip model.
Claim 18
Dependent Claim
21 words
The apparatus according to claim 15, wherein the reconstructed 2 dimension lip model is reconstructed based on 7 basic voice syllables.
Claim 19
Dependent Claim
18 words
The apparatus according to claim 16, wherein the rotation angle is calculated by using an affine transform technique.
Claim 20
Dependent Claim
88 words
The apparatus according to claim 17, wherein the transmitting means includes: means for computing movement quantities within a predetermined computing range between the actual lip image and either one of the reconstructed 2 dimension lip model in response to the first selection signal provided from the presenting means and the rotated 2 dimension lip model in response to the second selection signal provided from the presenting means to generate the lip movement parameters; and means for multiplexing the lip movement parameters, the voice signals and the rotation angle.
Claim 21
Dependent Claim
56 words
The apparatus according to claim 18, wherein the lip movement parameters include left-right movement parameters and up-down movement parameters of both left and right ending points of the 2 dimension lip model, and up-down movement parameters and forward-backward movement parameters of most upper and lower points in a central region of the 2 dimension lip model.
Claim 22
Dependent Claim
16 words
The apparatus according to claim 19, wherein the predetermined computing range is from -1 to 1.
References Cited (Backward Citations)
US4260229A
Cited
US5313522A
Cited
US5586215A
Cited
US5841470A
Cited
US5500673A
Cited
US5907351A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-01-10 | 2003-07-12 | 2004-01-11 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-01-10 | 2007-07-12 | 2008-01-11 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-01-10 | 2011-07-12 | 2012-01-11 | $7700.00 | Window Closed / Lapsed |