US5907351A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for cross-modal predictive coding for talking head sequences
Assignee: Alcatel Lucent SAS
•
Filing: 1995-10-24
•
Granted: 1999-05-25
•
Expires: 2015-10-24
Abstract
Abstract A method and apparatus for transmitting and remotely displaying the audio and visual portion of a person speaking so that the audio and visual signals are synchronized. The audio signal is constantly transmitted to the receiver and is also used to create a predicted image of the lips of the talking head. The actual lip image is compared to the predicted lip image. Based upon this comparison, it is determined which of three signals is to be transmitted to the receiver: no signal corresponding to the video signal, a signal corresponding only to the differences between the actual lip image and a predicted lip image, or the actual lip image. The receiver reconstructs a lip image based upon the audio signal received and the signal received, if any, corresponding to the video image and inserts it into the previously received video frame or modifies the previous frame accordingly.
Inventors
- 👤 Tsuhan Chen
- 👤 Ram R. Rao
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 16:29
CPC Classifications
H; H04; H04N; H04N21/00; H04N21/40; H04N21/43; H04N21/434; H04N21/4341; H04N19/00; H04N19/10; H04N19/134; H04N19/146; H04N19/147; H04N19/50; H04N19/503; H04N19/507; H04N21/20; H04N21/23; H04N21/233; H04N21/234; H04N21/23418; H04N21/235; H04N21/236; H04N21/2368; H04N21/24; H04N21/2402; H04N21/242; H04N21/4302; H04N21/4307; H04N21/43072; H04N21/435; H04N7/00; H04N7/14; H04N7/141; H04N7/142; G; G10; G10L; G10L21/00; G10L21/06; G10L21/10; G10L2021/105; H04N19/20
Claims Analysis
Total Claims: 40
Claim 1
Independent Claim
167 words
A method for remotely presenting a video output synchronized with an audio output, comprising: obtaining an actual video signal at a transmitter; obtaining an audio signal at the transmitter synchronized to the obtained actual video signal; processing at the transmitter the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; comparing at the transmitter the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; selecting, based upon predetermined criteria, a selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal; transmitting from the transmitter to a receiver the selected signal; generating the video output and the audio output by processing at the receiver the selected signal, the video output being synchronized with the audio output.
Claim 2
Independent Claim
167 words
A method for remotely presenting a video output synchronized with an audio output, comprising: obtaining an actual video signal at a transmitter; obtaining an audio signal at the transmitter synchronized to the obtained actual video signal; processing at the transmitter the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; comparing at the transmitter the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; selecting, based upon predetermined criteria, a selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal; transmitting from the transmitter to a receiver the selected signal; generating the video output and the audio output by processing at the receiver the selected signal, the video output being synchronized with the audio output.
Claim 3
Dependent Claim
24 words
The method of claim 1, wherein the predetermined criteria comprise an information transmission rate from the transmitter to the receiver of the selected signal.
Claim 4
Dependent Claim
25 words
The method of claim 1, wherein the predetermined criteria comprise distortion between a video image generated from the actual video signal and the video output.
Claim 5
Dependent Claim
25 words
The method of claim 3, wherein the predetermined criteria further comprise an information transmission rate from the transmitter to the receiver of the selected signal.
Claim 6
Dependent Claim
33 words
The method of claim 4, wherein the predetermined criteria is a cost function defined as C=R+(w×D), where R is the information transmission rate, D is the distortion, and w is a weighting factor.
Claim 7
Dependent Claim
27 words
The method of claim 5, wherein the signal selected in said selecting step is selected so as to result in the lowest value for the cost function.
Claim 8
Dependent Claim
40 words
The method of claim 1, wherein the actual video signal obtained in said obtaining step comprises lips of a person, and the portion of the obtained actual video signal compared in said comparing step comprises the lips of the person.
Claim 9
Dependent Claim
20 words
The method of claim 1, wherein said processing step at the transmitter comprises separating the obtained audio signal into phonemes.
Claim 10
Dependent Claim
20 words
The method of claim 8, wherein said generating step at the receiver comprises separating the obtained audio signal into phonemes.
Claim 11
Dependent Claim
23 words
The method of claim 8, wherein said processing step at the transmitter further comprises generating a shape of lips corresponding to the phonemes.
Claim 12
Dependent Claim
23 words
The method of claim 9, wherein said generating step at the receiver further comprises generating a shape of lips corresponding to the phonemes.
Claim 13
Dependent Claim
23 words
The method of claim 10, wherein said generating step at the receiver further comprises generating a shape of lips corresponding to the phonemes.
Claim 14
Dependent Claim
24 words
The method of claim 1, wherein the predetermined criteria comprise the magnitude of differences between the actual video signal and the predicted video signal.
Claim 15
Dependent Claim
101 words
The method of claim 13, wherein (a) the obtained audio signal and the obtained actual video signal is selected in said selecting step if the magnitude of differences is not less than a first predetermined threshold value, (b) the obtained audio signal and the difference video signal is selected in said selecting step if the magnitude of differences is less than the first predetermined threshold value and not less than a second predetermined threshold value, and (c) the obtained audio signal is selected in said selecting step if the magnitude of the differences is less than the second predetermined threshold value.
Claim 16
Independent Claim
146 words
A method for selecting a transmission signal for use in remotely presenting a video output synchronized with an audio output, the video and audio outputs being generated from the transmission signal, comprising: obtaining an actual video signal; obtaining an audio signal synchronized to the obtained actual video signal; processing the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; comparing the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; selecting, based upon predetermined criteria, a selected signal to comprise the transmission signal, the selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal.
Claim 17
Independent Claim
146 words
A method for selecting a transmission signal for use in remotely presenting a video output synchronized with an audio output, the video and audio outputs being generated from the transmission signal, comprising: obtaining an actual video signal; obtaining an audio signal synchronized to the obtained actual video signal; processing the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; comparing the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; selecting, based upon predetermined criteria, a selected signal to comprise the transmission signal, the selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal.
Claim 18
Dependent Claim
18 words
The method of claim 15, wherein the predetermined criteria comprise an information transmission rate of the transmission signal.
Claim 19
Dependent Claim
30 words
The method of claim 15, wherein the predetermined criteria comprise distortion between a video image generated from the actual video signal and the video output generated from the transmission signal.
Claim 20
Dependent Claim
19 words
The method of claim 17, wherein the predetermined criteria further comprise an information transmission rate of the transmission signal.
Claim 21
Dependent Claim
33 words
The method of claim 18, wherein the predetermined criteria is a cost function defined as C=R+(w×D), where R is the information transmission rate, D is the distortion, and w is a weighting factor.
Claim 22
Dependent Claim
27 words
The method of claim 19, wherein the signal selected in said selecting step is selected so as to result in the lowest value for the cost function.
Claim 23
Dependent Claim
40 words
The method of claim 15, wherein the actual video signal obtained in said obtaining step comprises lips of a person, and the portion of the obtained actual video signal compared in said comparing step comprises the lips of the person.
Claim 24
Dependent Claim
17 words
The method of claim 15, wherein said processing step comprises separating the obtained audio signal into phonemes.
Claim 25
Dependent Claim
20 words
The method of claim 22, wherein said processing step further comprises generating a shape of lips corresponding to the phonemes.
Claim 26
Dependent Claim
24 words
The method of claim 15, wherein the predetermined criteria comprise the magnitude of differences between the actual video signal and the predicted video signal.
Claim 27
Dependent Claim
101 words
The method of claim 24, wherein (a) the obtained audio signal and the obtained actual video signal is selected in said selecting step if the magnitude of differences is not less than a first predetermined threshold value, (b) the obtained audio signal and the difference video signal is selected in said selecting step if the magnitude of differences is less than the first predetermined threshold value and not less than a second predetermined threshold value, and (c) the obtained audio signal is selected in said selecting step if the magnitude of the differences is less than the second predetermined threshold value.
Claim 28
Independent Claim
156 words
An apparatus for selecting a transmission signal for use in remotely presenting a video output synchronized with an audio output, the video and audio outputs being generated from the transmission signal, comprising: means for obtaining an actual video signal; means for obtaining an audio signal synchronized to the obtained actual video signal; means for processing the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; means for comparing the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; means for selecting, based upon predetermined criteria, a selected signal to comprise the transmission signal, the selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal.
Claim 29
Independent Claim
156 words
An apparatus for selecting a transmission signal for use in remotely presenting a video output synchronized with an audio output, the video and audio outputs being generated from the transmission signal, comprising: means for obtaining an actual video signal; means for obtaining an audio signal synchronized to the obtained actual video signal; means for processing the obtained audio signal to generate a predicted video signal corresponding to the obtained audio signal; means for comparing the predicted video signal to at least a portion of the obtained actual video signal to determine differences therebetween; means for selecting, based upon predetermined criteria, a selected signal to comprise the transmission signal, the selected signal comprising one of (a) the obtained audio signal and the obtained actual video signal, (b) the obtained audio signal and a difference video signal representative of the differences between the obtained actual video signal and the predicted video signal, and (c) the obtained audio signal.
Claim 30
Dependent Claim
18 words
The apparatus of claim 26, wherein the predetermined criteria comprise an information transmission rate of the transmission signal.
Claim 31
Dependent Claim
30 words
The apparatus of claim 26, wherein the predetermined criteria comprise distortion between a video image generated from the actual video signal and the video output generated from the transmission signal.
Claim 32
Dependent Claim
19 words
The apparatus of claim 28, wherein the predetermined criteria further comprise an information transmission rate of the transmission signal.
Claim 33
Dependent Claim
33 words
The apparatus of claim 29, wherein the predetermined criteria is a cost function defined as C=R+(w×D), where R is the information transmission rate, D is the distortion, and w is a weighting factor.
Claim 34
Dependent Claim
27 words
The apparatus of claim 30, wherein the signal selected by said selecting means is selected so as to result in the lowest value for the cost function.
Claim 35
Dependent Claim
40 words
The apparatus of claim 26, wherein the actual video signal obtained by said obtaining means comprises lips of a person, and the portion of the obtained actual video signal compared in said comparing means comprises the lips of the person.
Claim 36
Dependent Claim
16 words
The apparatus of claim 26, wherein said processing means separates the obtained audio signal into phonemes.
Claim 37
Dependent Claim
18 words
The apparatus of claim 33, wherein said processing means generates a shape of lips corresponding to the phonemes.
Claim 38
Dependent Claim
24 words
The apparatus of claim 26, wherein the predetermined criteria comprise the magnitude of differences between the actual video signal and the predicted video signal.
Claim 39
Dependent Claim
101 words
The apparatus of claim 35, wherein (a) the obtained audio signal and the obtained actual video signal is selected by said selecting means if the magnitude of differences is not less than a first predetermined threshold value, (b) the obtained audio signal and the difference video signal is selected by said selecting means if the magnitude of differences is less than the first predetermined threshold value and not less than a second predetermined threshold value, and (c) the obtained audio signal is selected by said selecting means if the magnitude of the differences is less than the second predetermined threshold value.
Claim 40
Dependent Claim
39 words
The apparatus of claim 26, further comprising a receiving means for receiving the transmission signal and for generating and presenting the video output synchronized with the audio output in response to the transmission signal received by said receiving means.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-05-24 | 2002-11-23 | 2003-05-25 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-05-24 | 2006-11-23 | 2007-05-25 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-05-24 | 2010-11-23 | 2011-05-25 | $7700.00 | Window Closed / Lapsed |