US5586215A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Neural network acoustic and visual speech recognition system
Assignee: Ricoh Co Ltd
•
Filing: 1992-05-26
•
Granted: 1996-12-17
•
Expires: 2012-05-26
Abstract
Abstract The apparatus for the recognition of speech comprises an acoustic preprocessor, a visual preprocessor, and a speech classifier that operates on the acoustic and visual preprocessed data. The acoustic preprocessor comprises a log mel spectrum analyzer that produces an equal mel bandwidth log power spectrum. The visual processor detects the motion of a set of fiducial markers on the speaker's face and extracts a set of normalized distance vectors describing lip and mouth movement. The speech classifier uses a multilevel time-delay neural network operating on the preprocessed acoustic and visual data to form an output probability distribution that indicates the probability of each candidate utterance having been spoken, based on the acoustic and visual data.
Inventors
- 👤 David G. Stork
- 👤 Gregory J. Wolff
- 👤 Earl I. Levine
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-03 16:49
CPC Classifications
G; G06; G06N; G06N3/00; G06N3/02; G06N3/04; G06N3/049; G06F; G06F18/00; G06F18/20; G06F18/25; G06F18/254; G06F18/256; G06V; G06V30/00; G06V30/10; G06V30/24; G06V30/248; G06V40/00; G06V40/10; G06V40/16; G06V40/168; G10; G10L; G10L15/00; G10L15/08; G10L15/16; G10L15/24; G10L15/25; G10L15/02; G10L25/00; G10L25/03; G10L25/18
Claims Analysis
Total Claims: 18
Claim 1
Independent Claim
145 words
A speech recognition system for recognizing utterances belonging to a pre-established set of allowable candidate utterances using acoustic speech signals and selected concomitant dynamic visual facial feature motion between selected facial features associated with acoustic speech generation, comprising: a. an acoustic feature extraction apparatus for converting signals representative of acoustic speech into a corresponding acoustic feature vector set of signals; b. a dynamic visual feature extraction apparatus for converting signals representative of the selected concomitant dynamic visual facial feature motion associated with acoustic speech generation into a corresponding visual feature vector set of signals; and c. a time delay neural network classifying apparatus for generating a conditional probability distribution of the allowable candidate speech utterances by accepting and operating on a set of current and time delayed dynamic acoustic feature and visual feature vector sets respectively supplied by the acoustic and visual feature extraction apparatus.
Claim 2
Dependent Claim
145 words
A speech recognition system for recognizing utterances belonging to a pre-established set of allowable candidate utterances using acoustic speech signals and selected concomitant dynamic visual facial feature motion between selected facial features associated with acoustic speech generation, comprising: a. an acoustic feature extraction apparatus for converting signals representative of acoustic speech into a corresponding acoustic feature vector set of signals; b. a dynamic visual feature extraction apparatus for converting signals representative of the selected concomitant dynamic visual facial feature motion associated with acoustic speech generation into a corresponding visual feature vector set of signals; and c. a time delay neural network classifying apparatus for generating a conditional probability distribution of the allowable candidate speech utterances by accepting and operating on a set of current and time delayed dynamic acoustic feature and visual feature vector sets respectively supplied by the acoustic and visual feature extraction apparatus.
Claim 3
Dependent Claim
19 words
The speech recognition system of claim 1 wherein the acoustic feature extraction apparatus comprises a logarithmic power spectrum analyzer.
Claim 4
Dependent Claim
16 words
The speech recognition system of claim 2 wherein the spectrum analyzer is a non-uniform bandwidth analyzer.
Claim 5
Dependent Claim
18 words
The speech recognition system of claim 3 wherein the spectrum analyzer is a uniform mel bandwidth spectrum analyzer.
Claim 6
Dependent Claim
27 words
The speech recognition system of claim 2 wherein the acoustic feature extraction apparatus further comprises a microphone and signal conditioning apparatus for acquisition of acoustic speech signals.
Claim 7
Dependent Claim
53 words
The speech recognition system of claim 1 wherein the dynamic visual feature extraction apparatus further comprises transducer means for sensing of motion between the selected facial features and for converting the sensed motion into an electrical signal representative of the motion for use in generating the visual dynamic feature vector set of signals.
Claim 8
Dependent Claim
34 words
The speech recognition system of claim 6 wherein the sensed selected facial features comprise upper lip motion relative to lower lip motion, relative mouth corner to corner motion, and relative nose to chin motion.
Claim 9
Dependent Claim
67 words
The speech recognition system of claim 7 wherein the visual dynamic feature extraction apparatus comprises: a. a video camera for producing a dynamic pixel image representation of the mouth and lips of the utterance speaker; and b. means for characterizing mouth shape from video frame images produced by the video camera by measuring a multiplicity of distances between the shadows associated with the upper and lower lips.
Claim 10
Dependent Claim
37 words
The speech recognition system of claim 7 wherein the facial motion deviations are normalized for facial size and relative lip motion deviations are additionally normalized by removing a lip separation distance associated with a non-speech-generating rest condition.
Claim 11
Dependent Claim
155 words
The speech recognition system of claim 1 wherein the time delay neural network classifying apparatus comprises: a. a tapped delay line input layer for accepting a sequence of acoustic and visual time-varying feature vectors and for providing a multiplicity of sequential acoustic and visual feature vectors in parallel at the delay line output taps: b. a hidden layer of neural cells coupled to the output taps of the tapped delay line for enhancement of time dependent features; c. a classification layer of neural cells coupled to the output of the hidden layer neural cells for generating a set of output time varying signals each representative of the probability of its corresponding utterance being present; and d. an averaging layer apparatus, connected to the output of the classification layer, for generating a set of time averaged varying outputs, one for each allowable utterance type, representative of a combined conditional probability that the associated utterance was spoken.
Claim 12
Dependent Claim
43 words
The speech recognition system of claim 10 wherein the neural network classifying apparatus further comprises means, connected to the outputs of the averaging layer apparatus, for normalizing the outputs representative of the combined conditional probabilities, so that all probabilities sum up to one.
Claim 13
Dependent Claim
23 words
The speech recognition system of claim 10 wherein the hidden layer comprises separate acoustic and visual neural network time dependent feature extraction apparatus.
Claim 14
Dependent Claim
133 words
The speech recognition system of claim 12 wherein: a. the classification layer further comprises separate acoustic and visual neural network classification apparatus, each coupled only to their respective acoustic and visual hidden layer feature extraction apparatus for generating a separate set of acoustic and visual time varying output signals, each set representative of a conditional probability that an associated utterance was spoken; b. the averaging layer further comprises separate acoustic and visual averaging means for generating time averaged varying outputs representative of a conditional probability that the associated utterance was spoken; and c. the averaging layer further comprises a probability combining means for forming an estimate of the utterance joint conditional probability distribution by multiplying the separate corresponding pairs of probabilities generated at the output of the separate acoustic and visual classification apparatus.
Claim 15
Dependent Claim
41 words
The speech recognition system of claim 1 further comprising means for selecting a candidate utterance with the highest conditional probability as the most probable candidate speech utterance from the conditional probability distribution generated by the time delay neural network classifying apparatus.
Claim 16
Dependent Claim
106 words
A speech recognition system for recognizing utterances belonging to a pre-established set of allowable candidate utterances, comprising: a. a visual feature extraction apparatus for converting signals representative of facial features associated with acoustic speech generation into a corresponding set of visual feature vector signals at uniform time intervals at its output; and b. a time delay neural network classifying apparatus connected to the output of the visual feature extraction apparatus, having means for generating a conditional probability distribution of the allowable candidate utterances from the set of visual feature vector signals produced by the visual feature extraction apparatus and generated over a prescribed interval of time.
Claim 17
Dependent Claim
106 words
A speech recognition system for recognizing utterances belonging to a pre-established set of allowable candidate utterances, comprising: a. a visual feature extraction apparatus for converting signals representative of facial features associated with acoustic speech generation into a corresponding set of visual feature vector signals at uniform time intervals at its output; and b. a time delay neural network classifying apparatus connected to the output of the visual feature extraction apparatus, having means for generating a conditional probability distribution of the allowable candidate utterances from the set of visual feature vector signals produced by the visual feature extraction apparatus and generated over a prescribed interval of time.
Claim 18
Dependent Claim
144 words
The speech recognition system of claim 15 wherein the time delay neural network classifying apparatus comprises: a. a tapped delay line input layer for accepting a sequence of visual feature vectors and for providing a multiplicity of sequential visual vectors in parallel at the delay line output taps: b. a hidden layer of neural cells coupled to the output taps of the delay line for enhancement of time dependent features; c. a classification layer coupled to the output of the hidden layer neural cells for generating a set of output time varying signals each representative of the probability of its corresponding utterance being present; and d. an averaging layer apparatus, connected to the output of the classification layer, for generating a set of time averaged varying outputs, one for each allowable utterance type, representative of a conditional probability that the associated utterance was spoken.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1999-12-17 | 2000-06-17 | 2000-12-17 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2003-12-17 | 2004-06-17 | 2004-12-17 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2007-12-17 | 2008-06-17 | 2008-12-17 | $7700.00 | Window Closed / Lapsed |