US6567775B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Fusion of audio and video based speaker identification for multimedia information access
Assignee: International Business Machines Corp
•
Filing: 2000-04-26
•
Granted: 2003-05-20
•
Expires: 2020-04-26
Abstract
Abstract A method and apparatus are disclosed for identifying a speaker in an audio-video source using both audio and video information. An audio-based speaker identification system identifies one or more potential speakers for a given segment using an enrolled speaker database. A video-based speaker identification system identifies one or more potential speakers for a given segment using a face detector/recognizer and an enrolled face database. An audio-video decision fusion process evaluates the individuals identified by the audio-based and video-based speaker identification systems and determines the speaker of an utterance in accordance with the present invention. A linear variation is imposed on the ranked-lists produced using the audio and video information. The decision fusion scheme of the present invention is based on a linear combination of the audio and the video ranked-lists. The line with the higher slope is assumed to convey more discriminative information. The normalized slopes of the two lines are used as the weight of the respective results when combining the scores from the audio-based and video-based speaker analysis. In this manner, the weights are derived from the data itself.
Inventors
- 👤 Fereydoun Maali
- 👤 Mahesh Viswanathan
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 20:10
CPC Classifications
G; G10; G10L; G10L17/00; G10L17/06; G10L17/10; G06; G06F; G06F18/00; G06F18/20; G06F18/25; G06F18/254; G06F18/256; G06V; G06V20/00; G06V20/40
Claims Analysis
Total Claims: 20
Claim 1
Independent Claim
102 words
A method for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said method comprising the steps of: processing said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; processing said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and identifying said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 2
Independent Claim
102 words
A method for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said method comprising the steps of: processing said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; processing said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and identifying said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 3
Dependent Claim
31 words
The method of claim 1 , further comprising the step of imposing a linear variation on a list of said plurality of potential speakers and a list of said potential individuals.
Claim 4
Dependent Claim
43 words
The method of claim 2 , wherein said step of step of imposing a linear variation further comprises the steps of removing outliers using a Hough transform and fitting the surviving points set to a line using a least mean square error method.
Claim 5
Dependent Claim
72 words
The method of claim 3 , further comprising the step of representing said list of potential speakers as a straight line having a slope, m 1 , and said list of potential individuals as a straight line having a slope, m 2 , and wherein said slopes of said lines when normalized by their sum are used as the weight of the said audio and video information, respectively, during said identifying step.
Claim 6
Dependent Claim
98 words
The method of claim 4 , wherein said identifying step further comprises the step of computing a fused score, FS k , for each speaker as follows: FS k =W 1 ( m 1 ×rank k +b 1 ) + w 2 ( m 2×rank k +b 2 ) where rank k is the rank for speaker k, and w 1 and w 2 represent said audio and video information,respectively, as follows: w 1 = m 1 m 1 + m 2 and w 2 = m 2 m 1 + m 2 .
Claim 7
Independent Claim
102 words
A method for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said method comprising the steps of: processing said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; processing said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and identifying said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 8
Independent Claim
102 words
A method for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said method comprising the steps of: processing said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; processing said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and identifying said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 9
Dependent Claim
27 words
The method of claim 6 , further comprising the step of imposing a linear variation on said ranked-list of potential speakers and said ranked-list of potential individuals.
Claim 10
Dependent Claim
43 words
The method of claim 7 , wherein said step of step of imposing a linear variation further comprises the steps of removing outliers using a Hough transform and fitting the surviving points set to a line using a least mean square error method.
Claim 11
Dependent Claim
67 words
The method of claim 8 , further comprising the step of representing said ranked-list of potential speakers as a straight line having a slope, m 1 , and said ranked-list of potential individuals as a straight line having a slope, m 2 , and wherein said slopes of said lines are used as the weight of the said audio and video information, respectively, during said identifying step.
Claim 12
Dependent Claim
100 words
The method of claim 9 , wherein said identifying step further comprises the step of computing a fused score, FS k , for each speaker as follows: FS k =w 1 ( m 1 ×rank k +b 1 ) + w 2 ( m 2 ×rank k +b 2 ) where rank k is the rank for speaker k, and w 1 and w 2 represent said audio and video information, respectively, as follows: w 1 = m 1 m 1 + m 2 and w 2 = m 2 m 1 + m 2 .
Claim 13
Independent Claim
126 words
A system for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said system comprising: a memory that stores computer-readable code; and a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to: process said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; process said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 14
Independent Claim
126 words
A system for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said system comprising: a memory that stores computer-readable code; and a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to: process said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; process said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 15
Independent Claim
126 words
A system for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said system comprising: a memory that stores computer-readable code; and a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to: process said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; process said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 16
Independent Claim
126 words
A system for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said system comprising: a memory that stores computer-readable code; and a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to: process said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; process said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 17
Dependent Claim
130 words
An article of manufacture for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said article of manufacture comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to process said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; a step to process said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and a step to identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 18
Dependent Claim
130 words
An article of manufacture for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said article of manufacture comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to process said audio information to identify a plurality of potential speakers, each of said identified speakers having an associated confidence score; a step to process said video information to identify a plurality of potential individuals in an image, each of said identified individuals having an associated confidence score; and a step to identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 19
Dependent Claim
130 words
An article of manufacture for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said article of manufacture comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to process said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; a step to process said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and a step to identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
Claim 20
Dependent Claim
130 words
An article of manufacture for identifying a speaker in an audio-video source, said audio-video source having audio information and video information, said article of manufacture comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to process said audio information to identify a ranked-list of potential speakers, each of said identified speakers having an associated confidence score; a step to process said video information to identify a ranked-list of potential individuals in an image, each of said identified individuals having an associated confidence score; and a step to identify said speaker in said audio-video source based on said audio and video information, wherein said audio and video information is weighted based on slope information derived from said confidence scores.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-05-19 | 2006-11-18 | 2007-05-20 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-05-19 | 2010-11-18 | 2011-05-20 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-05-19 | 2014-11-18 | 2015-05-20 | $7700.00 | Window Closed / Lapsed |