US6816836B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for audio-visual speech detection and recognition
Assignee: Nuance Communications Inc
•
Filing: 2002-08-30
•
Granted: 2004-11-09
•
Expires: 2022-08-30
Abstract
Abstract Techniques for providing speech recognition comprise the steps of processing a video signal associated with an arbitrary content video source, processing an audio signal associated with the video signal, and recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.
Inventors
- 👤 Sankar Basu
- 👤 Philippe Christian de Cuetos
- 👤 Stephane Herman Maes
- 👤 Chalapathy Venkata Neti
- 👤 Andrew William Senior
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:33
CPC Classifications
G; G10; G10L; G10L15/00; G10L15/24; G10L15/25; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161; G06V40/20; G10L25/00; G10L25/78
Claims Analysis
Total Claims: 61
Claim 1
Independent Claim
63 words
A method of providing speech recognition, the method comprising the steps of: processing a video signal associated with an arbitrary content video source; processing an audio signal associated with the video signal; and recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.
Claim 2
Independent Claim
63 words
A method of providing speech recognition, the method comprising the steps of: processing a video signal associated with an arbitrary content video source; processing an audio signal associated with the video signal; and recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.
Claim 3
Dependent Claim
15 words
The method of claim 1 , wherein the audio signal is representative of conversational speech.
Claim 4
Dependent Claim
34 words
The method of claim 1 , wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more face candidates.
Claim 5
Dependent Claim
30 words
The method of claim 3 , wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected face candidates.
Claim 6
Dependent Claim
35 words
The method of claim 4 , wherein the video signal processing operation further comprises the step of detecting whether the one or more detected facial features are characteristic of respective face candidates in frontal poses.
Claim 7
Dependent Claim
23 words
The method of claim 5 , wherein at least one of face, facial feature and pose detection employ Fisher linear discriminant (FLD) analysis.
Claim 8
Dependent Claim
25 words
The method of claim 5 , wherein at least one of face, facial feature and pose detection employ a distance from face space (DFFS) measure.
Claim 9
Dependent Claim
43 words
The method of claim 3 , wherein the video signal processing operation comprises the step of extracting visual feature vectors from the video signal when a face is detected, the visual feature vectors representing one or more facial features of the detected face.
Claim 10
Dependent Claim
15 words
The method of claim 8 , wherein the visual feature vectors represent grey scale parameters.
Claim 11
Dependent Claim
14 words
The method of claim 8 , wherein the visual feature vectors represent geometric features.
Claim 12
Dependent Claim
18 words
The method of claim 8 , wherein the visual feature vectors are normalized with respect to pose estimates.
Claim 13
Dependent Claim
39 words
The method of claim 8 , wherein the audio signal processing operation comprises the step of extracting audio feature vectors from the associated audio signal, the audio feature vectors representing one or more acoustic features of the audio signal.
Claim 14
Dependent Claim
18 words
The method of claim 12 , wherein probabilities are assigned to phonemes based on the audio feature vectors.
Claim 15
Dependent Claim
22 words
The method of claim 13 , wherein probabilities are assigned to one of phonemes and visemes based on the visual feature vectors.
Claim 16
Dependent Claim
26 words
The method of claim 14 , wherein respective joint probabilities are formed from corresponding probabilities associated with the visual feature vectors and the audio feature vectors.
Claim 17
Dependent Claim
34 words
The method of claim 15 , wherein, in forming the joint probabilities, the corresponding probabilities associated with the visual feature vectors and the audio feature vectors are assigned weights based on a confidence measure.
Claim 18
Dependent Claim
24 words
The method of claim 16 , wherein the joint probabilities are used to search for an appropriate output signal representative of the audio signal.
Claim 19
Dependent Claim
23 words
The method of claim 17 , wherein the acoustic probabilities are used to search for a word hypothesis representative of the audio signal.
Claim 20
Dependent Claim
27 words
The method of claim 18 , wherein the word hypothesis is re-calculated using the visual probabilities to generate an appropriate output signal representative of the audio signal.
Claim 21
Dependent Claim
22 words
The method of claim 12 , wherein visual feature vectors are combined with corresponding audio feature vectors to form audio-visual feature vectors.
Claim 22
Dependent Claim
22 words
The method of claim 20 , wherein probabilities are assigned to one of phonemes and visemes based on the audio-visual feature vectors.
Claim 23
Dependent Claim
23 words
The method of claim 21 , wherein the probabilities are used to search for an appropriate output signal representative of the audio signal.
Claim 24
Dependent Claim
21 words
The method of claim 1 , wherein at least one of the video signal and the audio signal are compressed signals.
Claim 25
Dependent Claim
15 words
The method of claim 1 , wherein compressed signals are decompressed prior to processing operations.
Claim 26
Dependent Claim
16 words
The method of claim 1 , wherein the arbitrary content video source provides MPEG-2 standard signals.
Claim 27
Dependent Claim
29 words
The method of claim 1 , wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.
Claim 28
Dependent Claim
96 words
Apparatus for providing speech recognition, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with an arbitrary content video source; (ii) process an audio signal associated with the video signal; and (iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal; and memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.
Claim 29
Dependent Claim
96 words
Apparatus for providing speech recognition, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with an arbitrary content video source; (ii) process an audio signal associated with the video signal; and (iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal; and memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.
Claim 30
Dependent Claim
15 words
The apparatus of claim 27 , wherein the audio signal is representative of conversational speech.
Claim 31
Dependent Claim
34 words
The apparatus of claim 27 , wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more face candidates.
Claim 32
Dependent Claim
30 words
The apparatus of claim 29 , wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected face candidates.
Claim 33
Dependent Claim
35 words
The apparatus of claim 30 , wherein the video signal processing operation further comprises the step of detecting whether the one or more detected facial features are characteristic of respective face candidates in frontal poses.
Claim 34
Dependent Claim
23 words
The apparatus of claim 31 , wherein at least one of face, facial feature and pose detection employ Fisher linear discriminant (FLD) analysis.
Claim 35
Dependent Claim
25 words
The apparatus of claim 31 , wherein at least one of face, facial feature and pose detection employ a distance from face space (DFFS) measure.
Claim 36
Dependent Claim
43 words
The apparatus of claim 29 , wherein the video signal processing operation comprises the step of extracting visual feature vectors from the video signal when a face is detected, the visual feature vectors representing one or more facial features of the detected face.
Claim 37
Dependent Claim
15 words
The apparatus of claim 34 , wherein the visual feature vectors represent grey scale parameters.
Claim 38
Dependent Claim
14 words
The apparatus of claim 34 , wherein the visual feature vectors represent geometric features.
Claim 39
Dependent Claim
18 words
The apparatus of claim 34 , wherein the visual feature vectors are normalized with respect to pose estimates.
Claim 40
Dependent Claim
39 words
The apparatus of claim 34 , wherein the audio signal processing operation comprises the step of extracting audio feature vectors from the associated audio signal, the audio feature vectors representing one or more acoustic features of the audio signal.
Claim 41
Dependent Claim
18 words
The apparatus of claim 38 , wherein probabilities are assigned to phonemes based on the audio feature vectors.
Claim 42
Dependent Claim
22 words
The apparatus of claim 39 , wherein probabilities are assigned to one of phonemes and visemes based on the visual feature vectors.
Claim 43
Dependent Claim
26 words
The apparatus of claim 40 , wherein respective joint probabilities are formed from corresponding probabilities associated with the visual feature vectors and the audio feature vectors.
Claim 44
Dependent Claim
34 words
The apparatus of claim 41 , wherein, in forming the joint probabilities, the corresponding probabilities associated with the visual feature vectors and the audio feature vectors are assigned weights based on a confidence measure.
Claim 45
Dependent Claim
24 words
The apparatus of claim 42 , wherein the joint probabilities are used to search for an appropriate output signal representative of the audio signal.
Claim 46
Dependent Claim
23 words
The apparatus of claim 39 , wherein the acoustic probabilities are used to search for a word hypothesis representative of the audio signal.
Claim 47
Dependent Claim
27 words
The apparatus of claim 44 , wherein the word hypothesis is re-calculated using the visual probabilities to generate an appropriate output signal representative of the audio signal.
Claim 48
Dependent Claim
22 words
The apparatus of claim 38 , wherein visual feature vectors are combined with corresponding audio feature vectors to form audio-visual feature vectors.
Claim 49
Dependent Claim
22 words
The apparatus of claim 46 , wherein probabilities are assigned to one of phonemes and visemes based on the audio-visual feature vectors.
Claim 50
Dependent Claim
23 words
The apparatus of claim 47 , wherein the probabilities are used to search for an appropriate output signal representative of the audio signal.
Claim 51
Dependent Claim
21 words
The apparatus of claim 27 , wherein at least one of the video signal and the audio signal are compressed signals.
Claim 52
Dependent Claim
15 words
The apparatus of claim 27 , wherein compressed signals are decompressed prior to processing operations.
Claim 53
Dependent Claim
16 words
The apparatus of claim 27 , wherein the arbitrary content video source provides MPEG-2 standard signals.
Claim 54
Dependent Claim
29 words
The apparatus of claim 27 , wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.
Claim 55
Dependent Claim
37 words
The apparatus of claim 27 , wherein a speech detection decision is made using information from at least one of the video signal, the audio signal, and a correlation between the video signal and the audio signal.
Claim 56
Independent Claim
63 words
A method of providing speech recognition, the method comprising the steps of: processing an image signal associated with an arbitrary content image source; processing an audio signal associated with the image signal; and recognizing at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal.
Claim 57
Independent Claim
63 words
A method of providing speech recognition, the method comprising the steps of: processing an image signal associated with an arbitrary content image source; processing an audio signal associated with the image signal; and recognizing at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal.
Claim 58
Dependent Claim
96 words
Apparatus for providing speech recognition, the apparatus comprising: at least one processor operable to: (i) process an image signal associated with an arbitrary content image source, (ii) process an audio signal associated with the image signal, and (iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal; and memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.
Claim 59
Dependent Claim
96 words
Apparatus for providing speech recognition, the apparatus comprising: at least one processor operable to: (i) process an image signal associated with an arbitrary content image source, (ii) process an audio signal associated with the image signal, and (iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal; and memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.
Claim 60
Dependent Claim
65 words
Apparatus for providing speech recognition, the apparatus comprising: means for processing a video signal associated with an arbitrary content video source; means for processing an audio signal associated with the video signal; and means for recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.
Claim 61
Dependent Claim
65 words
Apparatus for providing speech recognition, the apparatus comprising: means for processing a video signal associated with an arbitrary content video source; means for processing an audio signal associated with the video signal; and means for recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-11-09 | 2008-05-10 | 2008-11-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-11-09 | 2012-05-10 | 2012-11-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-11-09 | 2016-05-10 | 2016-11-09 | $7700.00 | Window Closed / Lapsed |