Patent US6219640B1 successfully refreshed from Google Patents.
US6219640B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Methods and apparatus for audio-visual speaker recognition and utterance verification
Assignee: International Business Machines Corp
•
Filing: 1999-08-06
•
Granted: 2001-04-17
•
Expires: 2019-08-06
Abstract
Abstract Methods and apparatus for performing speaker recognition comprise processing a video signal associated with an arbitrary content video source and processing an audio signal associated with the video signal. Then, an identification and/or verification decision is made based on the processed audio signal and the processed video signal. Various decision making embodiments may be employed including, but not limited to, a score combination approach, a feature combination approach, and a re-scoring approach. In another aspect of the invention, a method of verifying a speech utterance comprises processing a video signal associated with a video source and processing an audio signal associated with the video signal. Then, the processed audio signal is compared with the processed video signal to determine a level of correlation between the signals. This is referred to as unsupervised utterance verification. In a supervised utterance verification embodiment, the processed video signal is compared with a script representing an audio signal associated with the video signal to determine a level of correlation between the signals.
Inventors
- 👤 Sankar Basu
- 👤 Homayoon S. M. Beigi
- 👤 Stephane Herman Maes
- 👤 Benoit Emmanuel Ghislain Maison
- 👤 Chalapathy Venkata Neti
- 👤 Andrew William Senior
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-08 16:48
CPC Classifications
G; G10; G10L; G10L15/00; G10L15/24; G10L15/25; G07; G07C; G07C9/00; G07C9/30; G07C9/32; G07C9/37; G06; G06F; G06F18/00; G06F18/20; G06F18/25; G06F18/254; G06F18/256; G06V; G06V10/00; G06V10/70; G06V10/77; G06V10/80; G06V10/809; G06V10/811; G06V40/00; G06V40/10; G06V40/16; G10L15/22; G10L2015/226
Claims Analysis
Total Claims: 69
Claim 1
Independent Claim
54 words
A method of performing speaker recognition, the method comprising the steps of: processing a video signal associated with an arbitrary content video source; processing an audio signal associated with the video signal; and making at least one of an identification and verification decision based on the processed audio signal and the processed video signal.
Claim 2
Independent Claim
54 words
A method of performing speaker recognition, the method comprising the steps of: processing a video signal associated with an arbitrary content video source; processing an audio signal associated with the video signal; and making at least one of an identification and verification decision based on the processed audio signal and the processed video signal.
Claim 3
Dependent Claim
33 words
The method of claim 1 , wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more faces.
Claim 4
Dependent Claim
29 words
The method of claim 2 , wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected faces.
Claim 5
Dependent Claim
22 words
The method of claim 3 , wherein at least one of face and facial feature detection employ Fisher linear discriminant (FLD) analysis.
Claim 6
Dependent Claim
24 words
The method of claim 3 , wherein at least one of face and facial feature detection employ a distance from face space (DFFS) measure.
Claim 7
Dependent Claim
31 words
The method of claim 3 , wherein the video signal processing operation further comprises the step of recognizing one or more faces from the detecting faces using the detected facial features.
Claim 8
Dependent Claim
29 words
The method of claim 6 , wherein the video signal processing operation further comprises the step of performing a confidence estimation procedure on results of the face recognition operation.
Claim 9
Dependent Claim
24 words
The method of claim 6 , wherein the audio signal processing operation comprises the step of recognizing a speaker associated with the audio signal.
Claim 10
Dependent Claim
30 words
The method of claim 8 , wherein the audio signal processing operation further comprises the step of performing a confidence estimation procedure on results of the audio speaker recognition operation.
Claim 11
Dependent Claim
33 words
The method of claim 8 , wherein respective results of the face recognition and audio speaker recognition operations are used to make at least one of the identification decision and the verification decision.
Claim 12
Dependent Claim
28 words
The method of claim 10 , wherein the results of one of the recognition operations are used to modify the results of the other of the recognition operations.
Claim 13
Dependent Claim
15 words
The method of claim 11 , wherein the decision is based on the modified results.
Claim 14
Dependent Claim
37 words
The method of claim 10 , wherein the results are combined such that one set of top N respective scores are generated for the face recognition and audio speaker recognition operations and used to make the decision.
Claim 15
Dependent Claim
25 words
The method of claim 10 , wherein the results include the top N respective scores generated during the face recognition and audio speaker recognition operations.
Claim 16
Dependent Claim
18 words
The method of claim 14 , wherein the top N respective scores are combined using a mixture parameter.
Claim 17
Dependent Claim
24 words
The method of claim 15 , wherein the mixture parameter is selected within a range which maximizes the highest and the second highest scores.
Claim 18
Dependent Claim
27 words
The method of claim 15 , wherein the mixture parameter is selected according to a reliability measure associated with the face recognition and audio speaker recognition operations.
Claim 19
Dependent Claim
23 words
The method of claim 17 , wherein the mixture parameter is optimized in accordance with a cost function representative of an error rate.
Claim 20
Dependent Claim
24 words
The method of claim 17 , wherein the mixture parameter is optimized in accordance with a cost function representative of a smoothed error rate.
Claim 21
Dependent Claim
21 words
The method of claim 1 , wherein at least one of the video signal and the audio signal are compressed signals.
Claim 22
Dependent Claim
15 words
The method of claim 1 , wherein compressed signals are decompressed prior to processing operations.
Claim 23
Dependent Claim
16 words
The method of claim 1 , wherein the arbitrary content video source provides MPEG-2 standard signals.
Claim 24
Dependent Claim
29 words
The method of claim 1 , wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.
Claim 25
Dependent Claim
25 words
The method of claim 1 , further comprising the step of enrolling a user in accordance with at least one of acoustic and visual information.
Claim 26
Dependent Claim
20 words
The method of claim 24 , wherein the result of the enrollment operation is a combined biometric representing multiple modalities.
Claim 27
Independent Claim
51 words
A method of verifying a speech utterance, the method comprising the steps of: processing a video signal associated with a video source; processing an audio signal associated with the video signal; and comparing the processed audio signal with the processed video signal to determine a level of correlation between the signals.
Claim 28
Independent Claim
51 words
A method of verifying a speech utterance, the method comprising the steps of: processing a video signal associated with a video source; processing an audio signal associated with the video signal; and comparing the processed audio signal with the processed video signal to determine a level of correlation between the signals.
Claim 29
Dependent Claim
25 words
The method of claim 26 , wherein the video signal processing operation further comprises the step of extracting visual feature vectors from the video signal.
Claim 30
Dependent Claim
24 words
The method of claim 27 , wherein the video signal processing operation further comprises the step of associating visemes with the extracted feature vectors.
Claim 31
Dependent Claim
36 words
The method of claim 28 , wherein the audio signal processing operation further comprises the step of extracting acoustic feature vectors and using the extracted features to generate a decoded script representative of the audio signal.
Claim 32
Dependent Claim
15 words
The method of claim 29 , wherein the decoded script is aligned with the visemes.
Claim 33
Dependent Claim
21 words
The method of claim 30 , wherein a likelihood of the alignment is computed and used to make the verification determination.
Claim 34
Independent Claim
49 words
A method of verifying a speech utterance, the method comprising the steps of: processing a video signal associated with a video source; and comparing the processed video signal with a script representing an audio signal associated with the video signal to determine a level of correlation between the signals.
Claim 35
Independent Claim
49 words
A method of verifying a speech utterance, the method comprising the steps of: processing a video signal associated with a video source; and comparing the processed video signal with a script representing an audio signal associated with the video signal to determine a level of correlation between the signals.
Claim 36
Dependent Claim
58 words
Apparatus for performing speaker recognition, the apparatus comprising: at least one processor operable to: process a video signal associated with an arbitrary content video source, (ii) process an audio signal associated with the video signal, and (iii) make at least one of an identification and verification decision based on the processed audio signal and the processed video signal.
Claim 37
Dependent Claim
58 words
Apparatus for performing speaker recognition, the apparatus comprising: at least one processor operable to: process a video signal associated with an arbitrary content video source, (ii) process an audio signal associated with the video signal, and (iii) make at least one of an identification and verification decision based on the processed audio signal and the processed video signal.
Claim 38
Dependent Claim
33 words
The apparatus of claim 33 , wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more faces.
Claim 39
Dependent Claim
29 words
The apparatus of claim 34 , wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected faces.
Claim 40
Dependent Claim
31 words
The apparatus of claim 35 , wherein the video signal processing operation further comprises the step of recognizing one or more faces from the detecting faces using the detected facial features.
Claim 41
Dependent Claim
24 words
The apparatus of claim 35 , wherein at least one of face and facial feature detection employ a distance from face space (DFFS) measure.
Claim 42
Dependent Claim
22 words
The apparatus of claim 35 , wherein at least one of face and facial feature detection employ Fisher linear discriminant (FLD) analysis.
Claim 43
Dependent Claim
29 words
The apparatus of claim 38 , wherein the video signal processing operation further comprises the step of performing a confidence estimation procedure on results of the face recognition operation.
Claim 44
Dependent Claim
24 words
The apparatus of claim 38 , wherein the audio signal processing operation comprises the step of recognizing a speaker associated with the audio signal.
Claim 45
Dependent Claim
30 words
The apparatus of claim 40 , wherein the audio signal processing operation further comprises the step of performing a confidence estimation procedure on results of the audio speaker recognition operation.
Claim 46
Dependent Claim
33 words
The apparatus of claim 40 , wherein respective results of the face recognition and audio speaker recognition operations are used to make at least one of the identification decision and the verification decision.
Claim 47
Dependent Claim
37 words
The apparatus of claim 42 , wherein the results are combined such that one set of top N respective scores are generated for the face recognition and audio speaker recognition operations and used to make the decision.
Claim 48
Dependent Claim
28 words
The apparatus of claim 42 , wherein the results of one of the recognition operations are used to modify the results of the other of the recognition operations.
Claim 49
Dependent Claim
15 words
The apparatus of claim 44 , wherein the decision is based on the modified results.
Claim 50
Dependent Claim
25 words
The apparatus of claim 42 , wherein the results include the top N respective scores generated during the face recognition and audio speaker recognition operations.
Claim 51
Dependent Claim
18 words
The apparatus of claim 46 , wherein the top N respective scores are combined using a mixture parameter.
Claim 52
Dependent Claim
24 words
The apparatus of claim 47 , wherein the mixture parameter is selected within a range which maximizes the highest and the second highest scores.
Claim 53
Dependent Claim
27 words
The apparatus of claim 47 , wherein the mixture parameter is selected according to a reliability measure associated with the face recognition and audio speaker recognition operations.
Claim 54
Dependent Claim
24 words
The apparatus of claim 49 , wherein the mixture parameter is optimized in accordance with a cost function representative of a smoothed error rate.
Claim 55
Dependent Claim
23 words
The apparatus of claim 49 , wherein the mixture parameter is optimized in accordance with a cost function representative of an error rate.
Claim 56
Dependent Claim
21 words
The apparatus of claim 33 , wherein at least one of the video signal and the audio signal are compressed signals.
Claim 57
Dependent Claim
15 words
The apparatus of claim 33 , wherein compressed signals are decompressed prior to processing operations.
Claim 58
Dependent Claim
16 words
The apparatus of claim 33 , wherein the arbitrary content video source provides MPEG-2 standard signals.
Claim 59
Dependent Claim
29 words
The apparatus of claim 33 , wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.
Claim 60
Dependent Claim
27 words
The apparatus of claim 33 , wherein the processor is further operable to enroll a user in accordance with at least one of acoustic and visual information.
Claim 61
Dependent Claim
20 words
The apparatus of claim 56 , wherein the result of the enrollment operation is a combined biometric representing multiple modalities.
Claim 62
Dependent Claim
56 words
Apparatus for verifying a speech utterance, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with a video source, (ii) process an audio signal associated with the video signal, and (iii) compare the processed audio signal with the processed video signal to determine a level of correlation between the signals.
Claim 63
Dependent Claim
56 words
Apparatus for verifying a speech utterance, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with a video source, (ii) process an audio signal associated with the video signal, and (iii) compare the processed audio signal with the processed video signal to determine a level of correlation between the signals.
Claim 64
Dependent Claim
53 words
Apparatus for verifying a speech utterance, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with a video source, and (ii) compare the processed video signal with a script representing an audio signal associated with the video signal to determine a level of correlation between the signals.
Claim 65
Dependent Claim
53 words
Apparatus for verifying a speech utterance, the apparatus comprising: at least one processor operable to: (i) process a video signal associated with a video source, and (ii) compare the processed video signal with a script representing an audio signal associated with the video signal to determine a level of correlation between the signals.
Claim 66
Independent Claim
54 words
A method of performing speaker recognition, the method comprising the steps of: processing an image signal associated with an arbitrary content image source; processing an audio signal associated with the image signal; and making at least one of an identification and verification decision based on the processed audio signal and the processed image signal.
Claim 67
Independent Claim
54 words
A method of performing speaker recognition, the method comprising the steps of: processing an image signal associated with an arbitrary content image source; processing an audio signal associated with the image signal; and making at least one of an identification and verification decision based on the processed audio signal and the processed image signal.
Claim 68
Dependent Claim
59 words
Apparatus for performing speaker recognition, the apparatus comprising: at least one processor operable to: (i) process an image signal associated with an arbitrary content image source, (ii) process an audio signal associated with the image signal, and (iii) make at least one of an identification and verification decision based on the processed audio signal and the processed image signal.
Claim 69
Dependent Claim
59 words
Apparatus for performing speaker recognition, the apparatus comprising: at least one processor operable to: (i) process an image signal associated with an arbitrary content image source, (ii) process an audio signal associated with the image signal, and (iii) make at least one of an identification and verification decision based on the processed audio signal and the processed image signal.
References Cited (Backward Citations)
US4449189A
Cited
US4757451A
Cited
US4845636A
Cited
US5412738A
Cited
US5625704A
Cited
US5602933A
Cited
US5897616A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2004-04-16 | 2004-10-16 | 2005-04-17 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2008-04-16 | 2008-10-16 | 2009-04-17 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2012-04-16 | 2012-10-16 | 2013-04-17 | $7700.00 | Window Closed / Lapsed |