US5572261A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Automatic audio to video timing measurement device and method
Assignee: Individual
•
Filing: 1995-06-07
•
Granted: 1996-11-05
•
Expires: 2015-06-07
Abstract
Abstract The apparatus and method described herein provides for detecting or measuring the relative timing of the audio and video portions of an audio visual program by inspection of the opening and closing of the mouth of a speaker, and comparing that opening and closing to the utterance of sounds associated therewith.
Inventors
- 👤 J. Carl Cooper
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-01 19:32
CPC Classifications
H; H04; H04N; H04N17/00
Claims Analysis
Total Claims: 34
Claim 1
Independent Claim
206 words
A method of determining the location and movement of the lips of a speaker in a sampled image, which said image may show one or more facial elements of said speaker including upper and lower lips, palate, teeth, tongue, and a dark hole between the lips, said method including the steps of: a) for each of a plurality of image frames, selecting a first group of samples from a known area of said image, b) inspecting at least one parameter of each of said samples of each said first group with respect to at least a first facial element, to identifying said samples which are taken from said first facial element, if any should exist, c) in response to step b), selecting from said identified samples a second group of samples, if any should exist, representing at least samples taken from said lips and said dark hole between the lips, d) determining the motion of said lips by inspecting said second group of samples corresponding to each said plurality of image frames to determine a sequence of change in at least said dark hole between the lips, e) determining said location and movement of the lips by comparing said sequence of change to known sequence patterns.
Claim 2
Independent Claim
206 words
A method of determining the location and movement of the lips of a speaker in a sampled image, which said image may show one or more facial elements of said speaker including upper and lower lips, palate, teeth, tongue, and a dark hole between the lips, said method including the steps of: a) for each of a plurality of image frames, selecting a first group of samples from a known area of said image, b) inspecting at least one parameter of each of said samples of each said first group with respect to at least a first facial element, to identifying said samples which are taken from said first facial element, if any should exist, c) in response to step b), selecting from said identified samples a second group of samples, if any should exist, representing at least samples taken from said lips and said dark hole between the lips, d) determining the motion of said lips by inspecting said second group of samples corresponding to each said plurality of image frames to determine a sequence of change in at least said dark hole between the lips, e) determining said location and movement of the lips by comparing said sequence of change to known sequence patterns.
Claim 3
Dependent Claim
29 words
A method as claimed in claim 1 wherein in said step c) samples representing at least one of the palate, teeth and tongue are included in said second group.
Claim 4
Dependent Claim
23 words
A method as claimed in claim 1 wherein said at least one parameter in step b) includes at least color saturation or hue.
Claim 5
Dependent Claim
28 words
A method as claimed in claim 1 wherein said inspecting at least one parameter in step b) includes inspecting in response to the color saturation, hue and brightness.
Claim 6
Independent Claim
139 words
A method for measuring the relative timing of an audio and a visual portion of a program including the steps of: a) identifying in said visual portion the presence and occurrence in time of known mouth sounds created by visual patterns of mouth movements, b) identifying in said audio portion the presence and occurrence in time of known mouth sounds created by audible patterns of mouth movements, c) storing first data representative of a sequence of said known mouth sounds of step a), d) storing second data representative of a sequence of said known mouth sounds of step b), e) comparing said first data of step c) with said second data of step d) to determine a relative position of a best match thereof, and measuring said relative timing in response to the relative position of the best match.
Claim 7
Independent Claim
139 words
A method for measuring the relative timing of an audio and a visual portion of a program including the steps of: a) identifying in said visual portion the presence and occurrence in time of known mouth sounds created by visual patterns of mouth movements, b) identifying in said audio portion the presence and occurrence in time of known mouth sounds created by audible patterns of mouth movements, c) storing first data representative of a sequence of said known mouth sounds of step a), d) storing second data representative of a sequence of said known mouth sounds of step b), e) comparing said first data of step c) with said second data of step d) to determine a relative position of a best match thereof, and measuring said relative timing in response to the relative position of the best match.
Claim 8
Dependent Claim
34 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by the presence of movements and nonmovements of the lips of a speaker portrayed by said visual portion.
Claim 9
Dependent Claim
34 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by parameters of the direction of movement of the lips of a speaker portrayed by said visual portion.
Claim 10
Dependent Claim
34 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by parameters of the lips getting closer or further apart of a speaker portrayed by said visual portion.
Claim 11
Dependent Claim
42 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by the mouth being open following the mouth opening and the mouth being closed following the mouth closing of a speaker portrayed by said visual portion.
Claim 12
Dependent Claim
42 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by analyzing said visual patterns of mouth movement to determine possible types of said mouth sounds being uttered by a speaker portrayed by said visual portion.
Claim 13
Dependent Claim
60 words
A method as claimed in claim 5 wherein said mouth sounds in step a) are determined by analyzing said visual patterns of mouth movement taken in conjunction with patterns of exposure of teeth and the dark hole formed by the open mouth to determine possible types of said mouth sounds being uttered by a speaker portrayed by said visual portion.
Claim 14
Dependent Claim
32 words
A method as claimed in claim 5 wherein said mouth sounds in step b) are determined by analyzing an amplitude related envelope of a band pass limited version of said audio portion.
Claim 15
Dependent Claim
39 words
A method as claimed in claim 5 wherein said mouth sounds in step b) are determined by comparing an amplitude related envelope of a band pass limited version of said audio portion to known envelopes of known mouth sounds.
Claim 16
Dependent Claim
47 words
A method as claimed in claim 5, 6, 7, 8, 9, 10, 11, 12 or 13 wherein said step for measuring is performed in response to uncorrected said audio portion and said video portion with an earlier arriving one thereof being delayed in response to said measuring.
Claim 17
Dependent Claim
57 words
A method as claimed in claim 5, 6, 7, 8, 9, 10, 11, 12 or 13 wherein an earlier arriving one of said audio portion and said video portion is delayed by said step for measuring which being performed in response the delayed one and a later arriving one of said audio portion and said video portion.
Claim 18
Independent Claim
204 words
An apparatus for determining the location and movement of the lips of a speaker in a sampled image, which said image may include facial elements the types of which may include ones of upper and lower lips, palate, teeth, tongue, and a dark hole between lips, said apparatus including: a) a selecting circuit to select a first group of samples from a known area of said image over a plurality of image frames, b) a first inspection circuit for inspecting at least one parameter of each of said samples of said first group to determine a type of facial element, if any, the samples are taken from, c) a storing circuit for storing a second group of samples representing at least samples taken from the lips and the dark hole between the lips in response to said first inspection circuit, d) a second inspection circuit for inspecting known ones of said second group of samples over a sequence of said image frames to determine a sequence of change in the type of facial element said known ones thereof are taken from, e) a comparison circuit for determining said location and movement of the lips by comparing said sequence of change to known sequence patterns.
Claim 19
Independent Claim
204 words
An apparatus for determining the location and movement of the lips of a speaker in a sampled image, which said image may include facial elements the types of which may include ones of upper and lower lips, palate, teeth, tongue, and a dark hole between lips, said apparatus including: a) a selecting circuit to select a first group of samples from a known area of said image over a plurality of image frames, b) a first inspection circuit for inspecting at least one parameter of each of said samples of said first group to determine a type of facial element, if any, the samples are taken from, c) a storing circuit for storing a second group of samples representing at least samples taken from the lips and the dark hole between the lips in response to said first inspection circuit, d) a second inspection circuit for inspecting known ones of said second group of samples over a sequence of said image frames to determine a sequence of change in the type of facial element said known ones thereof are taken from, e) a comparison circuit for determining said location and movement of the lips by comparing said sequence of change to known sequence patterns.
Claim 20
Dependent Claim
31 words
An apparatus as claimed in claim 16 wherein in said storing circuit, samples representing at least one of the palate, teeth and tongue are included in said second group of samples.
Claim 21
Dependent Claim
27 words
An apparatus as claimed in claim 16 wherein said at least one parameter in said first inspection circuit includes at least one of color saturation or hue.
Claim 22
Dependent Claim
30 words
An apparatus as claimed in claim 16 wherein said inspecting at least one parameter in said first inspection circuit includes inspecting in response to the color saturation, hue and brightness.
Claim 23
Independent Claim
158 words
An apparatus for measuring the relative timing of an audio and a visual portion of a program said apparatus including: a) a video identifying circuit responsive to said visual portion to identify the presence and occurrence in time of known mouth sounds, b) an audio identifying circuit responsive to said audio portion to identify the presence and occurrence in time of known mouth sounds, c) a first storing circuit for storing first information responsive to a sequence of said known mouth sounds of said video identifying circuit, d) a second storing circuit for storing second information responsive to a sequence of said known mouth sounds of said audio identifying circuit, e) a comparing circuit for comparing said first information of said first storing circuit with said second information of said second storing circuit to determine a relative position of a best match thereof, with the relative position of said best match being a measuring of said relative timing.
Claim 24
Independent Claim
158 words
An apparatus for measuring the relative timing of an audio and a visual portion of a program said apparatus including: a) a video identifying circuit responsive to said visual portion to identify the presence and occurrence in time of known mouth sounds, b) an audio identifying circuit responsive to said audio portion to identify the presence and occurrence in time of known mouth sounds, c) a first storing circuit for storing first information responsive to a sequence of said known mouth sounds of said video identifying circuit, d) a second storing circuit for storing second information responsive to a sequence of said known mouth sounds of said audio identifying circuit, e) a comparing circuit for comparing said first information of said first storing circuit with said second information of said second storing circuit to determine a relative position of a best match thereof, with the relative position of said best match being a measuring of said relative timing.
Claim 25
Dependent Claim
36 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by the presence of movements and nonmovements of the lips of a speaker portrayed by said visual portion.
Claim 26
Dependent Claim
36 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by parameters of the direction of movement of the lips of a speaker portrayed by said visual portion.
Claim 27
Dependent Claim
36 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by parameters of the lips of a speaker portrayed by said visual portion getting closer or further apart.
Claim 28
Dependent Claim
44 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by the mouth being open following the mouth opening and the mouth being closed following the mouth closing of a speaker portrayed by said visual portion.
Claim 29
Dependent Claim
40 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by analyzing patterns of mouth movement to determine possible types of sounds being uttered by a speaker portrayed by said visual portion.
Claim 30
Dependent Claim
57 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said video identifying circuit are determined by analyzing patterns of mouth movement taken in conjunction with patterns of exposure of teeth and black hole formed by an open mouth to determine possible types of sounds being uttered by a speaker portrayed by said visual portion.
Claim 31
Dependent Claim
33 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said audio identifying circuit are determined by analyzing an energy envelope of a band pass limited version of said audio portion.
Claim 32
Dependent Claim
40 words
An apparatus as claimed in claim 20 wherein said mouth sounds in said audio identifying circuit are determined by comparing an energy envelope of a band pass limited version of said audio portion to known envelopes of known mouth sounds.
Claim 33
Dependent Claim
49 words
An apparatus as claimed in claim 20, 21, 22, 23, 24, 25, 26, 27 or 28 further comprising a delay means, said delay means operates in response to uncorrected said audio portion and said video portion with an earlier arriving one thereof being delayed in response to said measuring.
Claim 34
Dependent Claim
59 words
An apparatus as claimed in claim 20, 21, 22, 23, 24, 25, 26, 27 or 28 further comprising a delay means, wherein an earlier arriving one of said audio portion and said video portion is delayed by said delay means operating in response the delayed one and a later arriving one of said audio portion and said video portion.
References Cited (Backward Citations)
US4313135A
Cited
US4313135B1
Cited
US5202761A
Cited
US4975960A
Cited
USRE33535E
Cited
US4841575A
Cited
US4769845A
Cited
US5387943A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1999-11-05 | 2000-05-06 | 2000-11-05 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2003-11-05 | 2004-05-06 | 2004-11-05 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2007-11-05 | 2008-05-06 | 2008-11-05 | $7700.00 | Window Closed / Lapsed |