US5550580A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Low bit rate audio-visual communication system having integrated perceptual speech and video coding
Assignee: Nokia of America Corp
•
Filing: 1995-05-31
•
Granted: 1996-08-27
•
Expires: 2015-05-31
Abstract
Abstract Disclosed is a low bit rate audio and video communication system which employs an integrated encoding system that dynamically allocates available bits among the audio and video signals to be encoded based on the content of the audio and video information and the manner in which the audio and video information will be perceived by a viewer. A dynamic bit allocation and encoding process will evaluate the current content of the audio and video information and allocate the available bits among the audio and video signals to be encoded. In addition, an appropriate audio encoding technique is dynamically selected based on the current content of the audio signal. A face location detection subroutine will detect and model the location of faces in each video frame, in order that the facial regions may be more accurately encoded than other portions of the video frame. A lip motion detection subroutine will detect the location and movement of the lips of a person present in a video scene, in order to determine when a person is speaking and to encode the lip regions more accurately. The audio and video signals generated by a second part to a communication are monitored to determine if the second party is paying attention to the audio and video information transmitted by the first party to the communication.
Inventors
- 👤 Yong Zhou
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-08 04:59
CPC Classifications
G; G10; G10L; G10L19/00; G10L19/012; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161; H; H04; H04N; H04N19/00; H04N19/10; H04N19/102; H04N19/132; H04N19/134; H04N19/136; H04N19/137; H04N19/14; H04N19/164; H04N19/169; H04N19/17; H04N19/176; H04N19/60; H04N19/61; H04N19/62; H04N19/63; H04N19/115; H04N19/146; H04N19/149; H04N19/15; H04N19/20
Claims Analysis
Total Claims: 13
Claim 1
Independent Claim
133 words
A method of selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said method comprising the steps of: analyzing said audio signal to determine if there is audio activity; analyzing said video signal to determine if said lips of said person in said view are moving; and encoding said audio signal with a speech specific audio encoding technique if said analyzing steps determine that said lips are moving while there is audio activity and encoding said audio signal with a non-speech specific audio encoding technique if said analyzing steps determine that said lips are not moving while there is audio activity.
Claim 2
Independent Claim
133 words
A method of selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said method comprising the steps of: analyzing said audio signal to determine if there is audio activity; analyzing said video signal to determine if said lips of said person in said view are moving; and encoding said audio signal with a speech specific audio encoding technique if said analyzing steps determine that said lips are moving while there is audio activity and encoding said audio signal with a non-speech specific audio encoding technique if said analyzing steps determine that said lips are not moving while there is audio activity.
Claim 3
Independent Claim
146 words
An apparatus for selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said apparatus comprising the steps of: means for analyzing said audio signal to determine if there is audio activity; means for analyzing said video signal to determine if said lips of said person in said view are moving; a speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are moving while there is audio activity; and a non-speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are not moving while there is audio activity.
Claim 4
Independent Claim
146 words
An apparatus for selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said apparatus comprising the steps of: means for analyzing said audio signal to determine if there is audio activity; means for analyzing said video signal to determine if said lips of said person in said view are moving; a speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are moving while there is audio activity; and a non-speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are not moving while there is audio activity.
Claim 5
Independent Claim
158 words
A method of selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said method comprising the steps of: analyzing said audio signal to determine if there is audio activity; analyzing said video signal to determine if said lips of said person in said view are moving: encoding said audio signal with a speech specific audio encoding technique if said analyzing steps determine that said lips are moving while there is audio activity and encoding said audio signal with a non-speech specific audio encoding technique if said analyzing steps determine that said lips are not moving while there is audio activity; and encoding said audio signal with a comfort noise encoding technique if said step of analyzing said audio signal determines that there is no audio activity.
Claim 6
Independent Claim
158 words
A method of selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said method comprising the steps of: analyzing said audio signal to determine if there is audio activity; analyzing said video signal to determine if said lips of said person in said view are moving: encoding said audio signal with a speech specific audio encoding technique if said analyzing steps determine that said lips are moving while there is audio activity and encoding said audio signal with a non-speech specific audio encoding technique if said analyzing steps determine that said lips are not moving while there is audio activity; and encoding said audio signal with a comfort noise encoding technique if said step of analyzing said audio signal determines that there is no audio activity.
Claim 7
Dependent Claim
23 words
The method of claim 3 wherein a bit rate of about 1 kilobits per second is used for the comfort noise encoding technique.
Claim 8
Independent Claim
171 words
An apparatus for selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said apparatus comprising the steps of: means for analyzing said audio signal to determine if there is audio activity; means for analyzing said video signal to determine if said lips of said person in said view are moving; a speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are moving while there is audio activity; a non-speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are not moving while there is audio activity; and a comfort noise audio encoder for encoding said audio signal if said means for analyzing said audio signal determines that there is no audio activity.
Claim 9
Independent Claim
171 words
An apparatus for selecting an audio encoding technique for encoding an audio signal by an audio-visual communication system that contains audio and video information, said video information including a view of a facial region of at least one person, said facial region including lips, said apparatus comprising the steps of: means for analyzing said audio signal to determine if there is audio activity; means for analyzing said video signal to determine if said lips of said person in said view are moving; a speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are moving while there is audio activity; a non-speech specific audio encoder for encoding said audio signal if said means for analyzing said audio and video signals determine that said lips are not moving while there is audio activity; and a comfort noise audio encoder for encoding said audio signal if said means for analyzing said audio signal determines that there is no audio activity.
Claim 10
Dependent Claim
26 words
The method of claim 1, wherein the step of analyzing said audio signals further comprises determining if the amount of audio energy exceeds a predetermined threshold.
Claim 11
Dependent Claim
58 words
The method of claim 1, wherein a predetermined limited number of bits are available for encoding both the audio and video information, and further comprising the step of dynamically allocating relatively few of the predetermined limited number of bits for encoding the audio signal when said analyzing steps determine that audio activity is not correlated with lip movement.
Claim 12
Dependent Claim
58 words
The method of claim 1, wherein a predetermined limited number of bits are available for encoding both the audio and video information, and further comprising the step of dynamically allocating a higher number of the predetermined limited number of bits for encoding the audio signal when said analyzing steps determine that audio activity is correlated with lip movement.
Claim 13
Dependent Claim
23 words
The method of claim 1 further comprising the step of dynamically allocating an appropriate number of bits for audio and video information encoding.
References Cited (Backward Citations)
US5426460A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1999-08-27 | 2000-02-26 | 2000-08-27 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2003-08-27 | 2004-02-26 | 2004-08-27 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2007-08-27 | 2008-02-26 | 2008-08-27 | $7700.00 | Window Closed / Lapsed |