Speaker recognition using spatiotemporal cues

Claim 1

A computer-implemented method for generating a feature vector for use in a person recognition system, the feature vector based on visual observations of a spoken utterance, the computer-implemented method comprising the computer-implemented steps of: (a) observing a spoken utterance by a person that is to be recognized; and (b) representing the person's mouth action by a cue that includes a lip separation distance function of time that is associated with the spoken utterance; (c) extracting a set of feature vector components from the cue including, (i) duration of visual utterance beginning at the lip separation onset and ending at the lip separation closure; (ii) maximum lip separation distance and time of occurrence, and (iii) location of a mouth opening centroid between the mouth's upper and lower lip.

Claim 11

A method for generating a feature vector for use in a person recognition system, the feature vector based on visual observations of a spoken utterance, the method comprising: (a) acquiring a sequence of video facial images that are representative of the person speaking an utterance; (b) forming a sequence of difference images from the sequence of video facial images, each difference image representing an absolute difference in gray scale between succeeding video facial images for detecting changing facial areas associated with the spoken utterance, including the video facial image mouth area; (c) generating a mask function image that defines a large contiguous area representing the video facial image mouth area of each video facial image because the video facial image mouth area exhibits most change during an utterance, by thresholding each difference image; (d) establishing a region of interest (ROI) for each video facial image by centering the region of interest at each mask function image centroid and encompassing each video facial image mouth area; (e) sampling each ROI gray-scale along a line passing through each centroid location parallel to a line of left-to-right facial symmetry, and producing a sampled ROI gray scale function representative of gray-scale values along a line through the center of the mouth area; (f) forming a spatiotemporal function of time by ordering each sampled ROI gray-scale function in a sequential order corresponding to the sequence of video facial images from which they were obtained; (g) gray-scale transforming the spatiotemporal (ST) function, for producing a speaker utterance cue-block, by comparing the ST function gray-scale values with a selected threshold value for extracting the ST function areas with darkest gray-scale corresponding to lower-lip to upper-lip gray-scale; and (h) extracting a feature vector from the speaker's utterance cue-block by determining a duration of the utterance beginning with initial lip separation and ending with lip closure, maximum lip-to-lip opening distance and time of maximum lip-to-lip opening.

Claim 12

A method for generating a feature vector for use in a person recognition system, the feature vector based on visual observations of a spoken utterance, the method comprising: (a) acquiring a sequence of video facial images that are representative of the person speaking an utterance; (b) forming a sequence of difference images from the sequence of video facial images, each difference image representing an absolute difference in gray scale between succeeding video facial images for detecting changing facial areas associated with the spoken utterance, including the video facial image mouth area; (c) generating a mask function image that defines a large contiguous area representing the video facial image mouth area of each video facial image because the video facial image mouth area exhibits most change during an utterance, by thresholding each difference image; (d) establishing a region of interest (ROI) for each video facial image by centering the region of interest at each mask function image centroid and encompassing each video facial image mouth area; (e) sampling each ROI gray-scale along a line passing through each centroid location parallel to a line of left-to-right facial symmetry, and producing a sampled ROI gray scale function representative of gray-scale values along a line through the center of the mouth area; (f) forming a spatiotemporal function of time by ordering each sampled ROI gray-scale function in a sequential order corresponding to the sequence of video facial images from which they were obtained; (g) gray-scale transforming the spatiotemporal (ST) function, for producing a speaker utterance cue-block, by comparing the ST function gray-scale values with a selected threshold value for extracting the ST function areas with darkest gray-scale corresponding to lower-lip to upper-lip gray-scale; and (h) extracting a feature vector from the speaker's utterance cue-block by determining a duration of the utterance beginning with initial lip separation and ending with lip closure, maximum lip-to-lip opening distance and time of maximum lip-to-lip opening.

Claim 30

A system-implemented method for performing recognition, comprising the system-implemented steps of: (a) visually observing a spoken utterance by a person that is to be recognized by use of a sequence of video images of the person's face; (b) extracting the person's mouth action from the sequence of video images by use of differenced video images produced by differencing sequential images; (c) detecting the person's mouth opening by thresholding the differenced video images and selecting a contiguous mouth region where gray-scale values represent a large change between sequential images, the contiguous mouth region forming a mask for extracting the person's mouth area from the sequence of video images; (d) locating the person's mouth area by calculating a centroid of the mouth area; (e) extracting a feature vector from the spoken utterance by determining a duration of the spoken utterance beginning at the lip separation onset and ending at the lip separation closure, lip-to-lip opening distance, and time of such a lip-to-lip opening; and (f) performing recognition by comparing the feature vector with a predetermined set of feature vectors.

Claim 31

A system-implemented method for performing recognition, comprising the system-implemented steps of: (a) visually observing a spoken utterance by a person that is to be recognized by use of a sequence of video images of the person's face; (b) extracting the person's mouth action from the sequence of video images by use of differenced video images produced by differencing sequential images; (c) detecting the person's mouth opening by thresholding the differenced video images and selecting a contiguous mouth region where gray-scale values represent a large change between sequential images, the contiguous mouth region forming a mask for extracting the person's mouth area from the sequence of video images; (d) locating the person's mouth area by calculating a centroid of the mouth area; (e) extracting a feature vector from the spoken utterance by determining a duration of the spoken utterance beginning at the lip separation onset and ending at the lip separation closure, lip-to-lip opening distance, and time of such a lip-to-lip opening; and (f) performing recognition by comparing the feature vector with a predetermined set of feature vectors.