Patent US2750489A successfully refreshed from Google Patents.
Patent US5596362A successfully refreshed from Google Patents.
US5596362A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Low bit rate audio-visual communication having improved face and lip region detection
Assignee: Nokia of America Corp
•
Filing: 1995-05-31
•
Granted: 1997-01-21
•
Expires: 2015-05-31
Abstract
Abstract Disclosed is a low bit rate audio and video communication system which employs an integrated encoding system that dynamically allocates available bits among the audio and video information and the manner in which the audio and video information will be perceived by a viewer. A dynamic bit allocation and encoding process will evaluate the current content of the audio and video information and allocate the available bits among the audio and video signals to be encoded based upon the perceptual significance of that information. A face location detection subroutine will detect and model the location of faces in each video frame, in order that the facial regions may be more accurately encoded than other portions of the video frame. A lip motion detection subroutine will detect the location and movement of the lips of a person present in a video scene, in order to determine when a person is speaking and to encode the lip and face regions more accurately. By way of example, the face region may be modelled as a first ellipse and the lip region may be modelled as a second smaller ellipse contained within the first ellipse.
Inventors
- 👤 Yong Zhou
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-30 15:23
CPC Classifications
G; G10; G10L; G10L19/00; G10L19/012; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161; H; H04; H04N; H04N19/00; H04N19/10; H04N19/102; H04N19/132; H04N19/134; H04N19/136; H04N19/137; H04N19/14; H04N19/164; H04N19/169; H04N19/17; H04N19/176; H04N19/60; H04N19/61; H04N19/62; H04N19/63; H04N19/115; H04N19/146; H04N19/149; H04N19/15; H04N19/20
Claims Analysis
Total Claims: 27
Claim 1
Independent Claim
139 words
A method for detecting lip motion in a video signal having a plurality of image frames which include a representation of a person's face, said video signal including motion information, said method comprising the steps of: performing a face location detection subroutine on said video signal in order to determine the location of said face in said video image frame; isolating said motion information found in said video signal; positioning a trial lip region in a plurality of locations in said detected face region; analyzing said isolated motion information to identify the trial lip region within said face region having the highest measured average intensity; concluding that said lips are moving if said highest measured average intensity exceeds a predefined threshold value; and utilizing the conclusion that said lips are moving to control subsequent processing of the video signal.
Claim 2
Independent Claim
139 words
A method for detecting lip motion in a video signal having a plurality of image frames which include a representation of a person's face, said video signal including motion information, said method comprising the steps of: performing a face location detection subroutine on said video signal in order to determine the location of said face in said video image frame; isolating said motion information found in said video signal; positioning a trial lip region in a plurality of locations in said detected face region; analyzing said isolated motion information to identify the trial lip region within said face region having the highest measured average intensity; concluding that said lips are moving if said highest measured average intensity exceeds a predefined threshold value; and utilizing the conclusion that said lips are moving to control subsequent processing of the video signal.
Claim 3
Dependent Claim
30 words
The method of claim 1 further comprising the step of encoding the lip region with greater accuracy if it is concluded that the lips in said lip region are moving.
Claim 4
Dependent Claim
28 words
The method of claim 1 further comprising the step of encoding the face with greater accuracy if it is concluded that the lips in said face are moving.
Claim 5
Dependent Claim
41 words
The method of claim 1 further comprising the steps of: creating a speaker data base; and storing an indication that a speaker is speaking in the speaker data base when it is concluded that the lips of the speaker are moving.
Claim 6
Dependent Claim
25 words
The method of claim 4 further comprising the step of: storing information relating to each person involved in a videoconference in the speaker data base.
Claim 7
Dependent Claim
37 words
The method of claim 1, wherein the step of: isolating said motion information further comprises the step of isolating the edges of an object which has moved from one image frame to the next sequential image frame.
Claim 8
Dependent Claim
36 words
The method of claim 1 further comprising an initialization step, including the step of prompting a party to a videoconference to manually identify the position of each speaker seen on a video screen by the party.
Claim 9
Dependent Claim
27 words
The method of claim 7 further comprising the step of: prompting the party to enter the number of faces seen on the video screen by the party.
Claim 10
Dependent Claim
25 words
The method of claim 8 further comprising the step of: creating a speaker data base with data for each speaker manually identified by the party.
Claim 11
Dependent Claim
32 words
The method of claim 9 further comprising the steps of: modelling the face with an ellipse characterized by ellipse characterizing data; and storing the ellipse characterizing data in said speaker data base.
Claim 12
Dependent Claim
30 words
The method of claim 7 further comprising the steps of: identifying the position of each speaker by pointing to each speaker; and clicking on each speaker after said pointing step.
Claim 13
Dependent Claim
23 words
The method of claim 1, wherein the face location subroutine includes a step of detecting the contrast between the face and its background.
Claim 14
Dependent Claim
30 words
The method of claim 1, wherein the face location subroutine includes an edge detection subroutine which performs the step of identifying edges at which a significant change in intensity occurs.
Claim 15
Dependent Claim
19 words
The method of claim 13, wherein the edge detection subroutine performs the further step of generating a diffence image.
Claim 16
Dependent Claim
17 words
The method of claim 14 further comprising the step of: comparing the difference image with a threshold.
Claim 17
Dependent Claim
41 words
The method of claim 14 further comprising the steps of: generating a nonzero segment corresponding to each edge identified during said step of identifying edges; and determining if at least one nonzero segment can be fitted within a face modelling ellipse.
Claim 18
Dependent Claim
29 words
The method of claim 1 further comprising the step of: initializing a speaker data base by automatically locating and characterizing the faces of all speakers involved in a videoconference.
Claim 19
Dependent Claim
32 words
The method of claim 17, wherein said step of performing a face location detection subroutine is facilitated by limiting searching to a search for only the faces characterized in said initializing step.
Claim 20
Dependent Claim
28 words
The method of claim 1, wherein said step of positioning a trial lip region begins by testing in a mouth area detected as having the highest motion activity.
Claim 21
Dependent Claim
31 words
The method of claim 1, wherein said step of positioning a trial lip region begins by analyzing corresponding speech sound and then predicting the position of the lips from said analysis.
Claim 22
Dependent Claim
26 words
The method of claim 1 further comprising the step of: storing indexed parameters of lip ellipses for various mouth positions correlating to various mouth produced sounds.
Claim 23
Independent Claim
157 words
A method for detecting lip motion in a video signal consisting of one or more image frames which include a representation of a person's face, said video signal including motion information, said method comprising the steps of: performing a face region location detection subroutine on said video signal in order to determine the location of said face in said video image frame, said face location detection subroutine including the step of modelling said face region as a first ellipse; isolating said motion information found in said video signal; positioning a trial lip region, modelled as a second ellipse which is smaller than and contained within the first ellipse, in a plurality of locations in said detected face region; analyzing said isolated motion information to identify the trial lip regioin within said face region having the highest measured average intensity; and concluding that said lips are moving if said highest measured average intensity exceeds a predefined threshold value.
Claim 24
Independent Claim
157 words
A method for detecting lip motion in a video signal consisting of one or more image frames which include a representation of a person's face, said video signal including motion information, said method comprising the steps of: performing a face region location detection subroutine on said video signal in order to determine the location of said face in said video image frame, said face location detection subroutine including the step of modelling said face region as a first ellipse; isolating said motion information found in said video signal; positioning a trial lip region, modelled as a second ellipse which is smaller than and contained within the first ellipse, in a plurality of locations in said detected face region; analyzing said isolated motion information to identify the trial lip regioin within said face region having the highest measured average intensity; and concluding that said lips are moving if said highest measured average intensity exceeds a predefined threshold value.
Claim 25
Dependent Claim
38 words
The method of claim 22 wherein a potential mouth area is defined in a lower half of the first ellipse, which potential mouth area is searched to identify the trial lip region having the highest measured average intensity.
Claim 26
Dependent Claim
39 words
The method of claim 22, wherein said ellipse modelling the face is characterized by a major axis a, a minor axis b, and angle alpha between the major axis a and a vertical line, and a center pixel value.
Claim 27
Dependent Claim
25 words
The method of claim 24 further comprising the step of: storing at least a portion of said ellipse characterizing data in a speaker data base.
References Cited (Backward Citations)
US4218704A
Cited
US4245248A
Cited
US4449189A
Cited
US5086480A
Cited
US5136659A
Cited
US4975969A
Cited
US4975978A
Cited
US5247590A
Cited
US5117287A
Cited
US5136374A
Cited
US5309232A
Cited
US5412738A
Cited
US5426460A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2000-01-21 | 2000-07-22 | 2001-01-21 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2004-01-21 | 2004-07-22 | 2005-01-21 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2008-01-21 | 2008-07-22 | 2009-01-21 | $7700.00 | Window Closed / Lapsed |