Patent US5771306A successfully refreshed from Google Patents.
US5771306A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for extracting speech related facial features for use in speech recognition systems
Assignee: Ricoh Co Ltd
•
Filing: 1993-10-22
•
Granted: 1998-06-23
•
Expires: 2013-10-22
Abstract
Abstract The apparatus for the recognition of speech comprises an acoustic preprocessor, a visual preprocessor, and a speech classifier that operates the acoustic and visual preprocessed data. The acoustic preprocessor comprises a log mel spectrum analyzer that produces an equal mel bandwidth log power spectrum. The visual processor detects the motion of a set of fiducial markers on the speaker's face and extracts a set of normalized distance vectors describing lip and mouth movement. The speech classifier uses a multilevel time-delay neural network operating on the preprocessed acoustic and visual data to form an output probability distribution that indicates the probability of each candidate utterance having been spoken, based on the acoustic and visual data.
Inventors
- 👤 David G. Stork
- 👤 Gregory Joseph Wolff
- 👤 Earl Isaac Levine
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 10:58
CPC Classifications
G; G06; G06N; G06N3/00; G06N3/02; G06N3/04; G06N3/049; G06F; G06F18/00; G06F18/20; G06F18/25; G06F18/254; G06F18/256; G06V; G06V30/00; G06V30/10; G06V30/24; G06V30/248; G06V40/00; G06V40/10; G06V40/16; G06V40/168; G10; G10L; G10L15/00; G10L15/08; G10L15/16; G10L15/24; G10L15/25; G10L15/02; G10L25/00; G10L25/03; G10L25/18
Claims Analysis
Total Claims: 12
Claim 1
Independent Claim
222 words
An apparatus for extracting a visual feature vector from a sequence of video camera images of frontal views of a speaker's face in a speech classification system, the apparatus comprising: a) a set of fiducial markers placed on a speaker's face in the vicinity of the lips, nose, and chin so that the fiducial markers are readily identifiable in a video camera image of the speaker's face, and the position and movement of the set of fiducial markers are presentative of physiological facial phenomena associated with speech generation; b) a video camera directed at the speaker's face for generating a sequence of electrical images of the speaker's face in the vicinity of the lips, nose, and chin; and c) a video processor for converting and storing the sequence of electrical images as a rectangular grid of digitized pixels and for detecting and locating a position for each member of the set of fiducial markers within each image of the sequence, the positions being elements of the visual feature vector that is independent of head shifts and rotations by establishing, as a reference vertical axis, a line connecting the centroids of the nose and the chin fiducial markers, and rotating all fiducial marker positions by an angle that is necessary to make the line connecting the nose and chin fiducial marker centroids vertical.
Claim 2
Independent Claim
222 words
An apparatus for extracting a visual feature vector from a sequence of video camera images of frontal views of a speaker's face in a speech classification system, the apparatus comprising: a) a set of fiducial markers placed on a speaker's face in the vicinity of the lips, nose, and chin so that the fiducial markers are readily identifiable in a video camera image of the speaker's face, and the position and movement of the set of fiducial markers are presentative of physiological facial phenomena associated with speech generation; b) a video camera directed at the speaker's face for generating a sequence of electrical images of the speaker's face in the vicinity of the lips, nose, and chin; and c) a video processor for converting and storing the sequence of electrical images as a rectangular grid of digitized pixels and for detecting and locating a position for each member of the set of fiducial markers within each image of the sequence, the positions being elements of the visual feature vector that is independent of head shifts and rotations by establishing, as a reference vertical axis, a line connecting the centroids of the nose and the chin fiducial markers, and rotating all fiducial marker positions by an angle that is necessary to make the line connecting the nose and chin fiducial marker centroids vertical.
Claim 3
Dependent Claim
52 words
The apparatus of claim 1 wherein the video processor locates the position of each member of the set of fiducial markers by detecting a set of pixels corresponding to each fiducial marker and then calculating a centroid for each set of pixels, the centroid being the position of the corresponding fiducial marker.
Claim 4
Dependent Claim
51 words
The apparatus of claim 1 wherein the set of fiducial markers are placed on the nose tip, the center of the chin, left and right mouth corners, and at least one vertically juxtaposed pair of fiducial markers, with one each marker on the upper and lower lips of the speaker's face.
Claim 5
Dependent Claim
50 words
The apparatus of claim 3 wherein the video processor further includes means for calculating a distance between the nose and chin fiducial marker positions, a distance between the left and right mouth corner fiducial marker positions, and a distance between each vertically juxtaposed pair on the upper and lower lips.
Claim 6
Independent Claim
209 words
A method for extracting a visual feature vector from a sequence of video camera images of frontal views of a speaker's face in a speech classification system, the method comprising the following steps: a) placing a set of fiducial markers on a speaker's face in the vicinity of the lips, nose, and chin so that the fiducial markers are readily identifiable in a video camera image of the speaker's face, and the movement and position of the set of fiducial markers are representative of physiological facial phenomena associated with speech generation; b) producing a sequence of raster scanned electrical video images of the speaker's face in the vicinity of the fiducial markers; c) sampling and quantizing each raster scanned video image so as to produce a grid of digitized pixels representative of each raster scanned video image; d) detecting a set of pixels representative of each fiducial marker; e) computing a location for each fiducial marker from each set of detected pixels associated with each fiducial marker; f) establishing a reference axis corresponding to a straight line passing through the location of the nose and chin fiducial markers: and g) rotating all fiducial maker positions by the angle required to rotate the reference axis to a true vertical orientation.
Claim 7
Independent Claim
209 words
A method for extracting a visual feature vector from a sequence of video camera images of frontal views of a speaker's face in a speech classification system, the method comprising the following steps: a) placing a set of fiducial markers on a speaker's face in the vicinity of the lips, nose, and chin so that the fiducial markers are readily identifiable in a video camera image of the speaker's face, and the movement and position of the set of fiducial markers are representative of physiological facial phenomena associated with speech generation; b) producing a sequence of raster scanned electrical video images of the speaker's face in the vicinity of the fiducial markers; c) sampling and quantizing each raster scanned video image so as to produce a grid of digitized pixels representative of each raster scanned video image; d) detecting a set of pixels representative of each fiducial marker; e) computing a location for each fiducial marker from each set of detected pixels associated with each fiducial marker; f) establishing a reference axis corresponding to a straight line passing through the location of the nose and chin fiducial markers: and g) rotating all fiducial maker positions by the angle required to rotate the reference axis to a true vertical orientation.
Claim 8
Dependent Claim
30 words
The method of claim 5 wherein the step of computing a location for each fiducial marker comprises computing a centroid for each set of pixels associated with each fiducial marker.
Claim 9
Dependent Claim
54 words
The method of claim 5 wherein the set of fiducial markers are placed on the nose tip, the center of the chin, the left and right mouth corners, and at least one vertically juxtaposed pair of fiducial markers with one fiducial marker placed on the upper lip and one placed on the lower lip.
Claim 10
Dependent Claim
52 words
The method of claim 7 further comprising the following steps: a) calculating a distance between the nose and chin fiducial marker; b) calculating a distance between the left and right mouth corner fiducial markers; and c) calculating a distance between each vertically juxtaposed fiducial marker of each pair placed on the lips.
Claim 11
Dependent Claim
31 words
The method of claim 7 wherein the set of vertically juxtaposed pairs of fiducial markers are placed at the lip center and midway between the lip center and each mouth corner.
Claim 12
Dependent Claim
129 words
The method of claim 9 further comprises the following steps for calculating elements of a visual feature vector: a) calculating a vertical distance between nose and chin fiducial marker positions; b) calculating a vertical distance between corresponding fiducial markers of each vertically juxtaposed fiducial marker pair; c) calculating a horizontal distance between upper lip fiducial marker positions located midway between the upper lip center and each mouth corner; d) calculating a horizontal distance between lower lip fiducial marker positions located midway between the lower lip center and each mouth corner; e) calculating a vertical distance between each fiducial marker of each vertically juxtaposed marker pair located midway between the lip centers and each mouth corner position; and f) calculating a horizontal distance between each mouth corner fiducial marker position.
References Cited (Backward Citations)
US4706296A
Cited
US4620286A
Cited
US4975960A
Cited
US4757541A
Cited
US4841575A
Cited
US4769845A
Cited
US4937872A
Cited
US5163111A
Cited
US5173945A
Cited
US5022089A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2001-06-22 | 2001-12-22 | 2002-06-23 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2005-06-22 | 2005-12-22 | 2006-06-23 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2009-06-22 | 2009-12-22 | 2010-06-23 | $7700.00 | Window Closed / Lapsed |