US5689575A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for processing images of facial expressions
Assignee: Rakuten Group Inc
•
Filing: 1994-11-21
•
Granted: 1997-11-18
•
Expires: 2014-11-21
Abstract
Abstract A real time output containing data relating to states of facial parts is generated. A facial area detection unit (11) has monitoring (51-57) and determining (59-61) processing circuits operating in a pipelining manner to determine position of the facial area. The monitoring circuits (51-57) monitor pixel value frequency using 3D histogram and backprojection processing. The generating facial area signal has masks applied by a unit (12) which supplies data to mouth area and eye area detection units (14,15). Each of these operate on similar principles to the facial area detection unit (11).
Inventors
- 👤 Hiroshi Sako
- 👤 Anthony Smith
- 👤 Mark Whitehouse
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 15:56
CPC Classifications
G; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161; G06V40/168
Claims Analysis
Total Claims: 21
Claim 1
Independent Claim
150 words
An image processing apparatus comprising: means for receiving a color facial image data stream in digital format; a plurality of image data processing circuits interconnected for real time processing the image data in a pipelining manner to provide a real time output, the circuits comprising: means for monitoring color values in the image data to identify image data pixels representing facial parts, and monitoring means comprising means for carrying out three-dimensional color histogram matching to monitor frequency of occurrence of pixel values in the image data and a backprojection means for comparing a generated histogram with a template histogram generated off-line, and means for determining states of the facial parts by monitoring positional coordinates of the identified pixels, said means for determining further comprising a counter for determining the area of the facial part or parts; and an output device outputting the determined facial part state data in real time.
Claim 2
Dependent Claim
150 words
An image processing apparatus comprising: means for receiving a color facial image data stream in digital format; a plurality of image data processing circuits interconnected for real time processing the image data in a pipelining manner to provide a real time output, the circuits comprising: means for monitoring color values in the image data to identify image data pixels representing facial parts, and monitoring means comprising means for carrying out three-dimensional color histogram matching to monitor frequency of occurrence of pixel values in the image data and a backprojection means for comparing a generated histogram with a template histogram generated off-line, and means for determining states of the facial parts by monitoring positional coordinates of the identified pixels, said means for determining further comprising a counter for determining the area of the facial part or parts; and an output device outputting the determined facial part state data in real time.
Claim 3
Dependent Claim
19 words
An apparatus as claimed in claim 1 further comprising means for normalizing the received color facial image data stream.
Claim 4
Dependent Claim
73 words
An apparatus as claimed in claim 1 wherein the processing circuits comprise: a facial area detection unit comprising circuits for carrying out facial area detection operations on the image data; and a facial part detection unit connected to said facial area detection unit and comprising processing circuits for determining states of a facial part within the detected facial area using the image data and an output signal from said facial area detection unit.
Claim 5
Dependent Claim
23 words
An apparatus as claimed in claim 3 further comprising a mask generating circuit connected between the facial area and facial part detection units.
Claim 6
Dependent Claim
24 words
An apparatus as claimed in claim 3 comprising two facial detection units, including: a mouth area detection unit; and an eye area detection unit.
Claim 7
Dependent Claim
47 words
An apparatus as claimed in claim 5 wherein the apparatus further comprises a mask generating circuit connected between the facial area detection unit and the facial part detection units, said mask generating circuit unit comprising means for generating an eye mask signal and a mouth mask signal.
Claim 8
Dependent Claim
30 words
An apparatus as claimed in claim 5 wherein the mouth area and eye area detection units are connected in the apparatus to operate on the same image data in parallel.
Claim 9
Dependent Claim
24 words
An apparatus as claimed in claim 1, wherein the apparatus further comprises means for carrying out a consistency check to validate determined positional data.
Claim 10
Dependent Claim
25 words
An apparatus as claimed in claim 8 wherein each processing circuit comprises means for carrying out a consistency check on positional data which it generates.
Claim 11
Dependent Claim
25 words
An apparatus as claimed in claim 8 wherein the consistency check means comprises a processor programmed to compare the positional data with reference positional data.
Claim 12
Dependent Claim
37 words
An apparatus as claimed in claim 1 wherein the monitoring means comprises means for carrying out binary erosion and binary dilation steps to generate image data in which noise is reduced and the subject area is recovered.
Claim 13
Dependent Claim
26 words
An apparatus as claimed in claim 1 further comprising image data processing circuits for determining location of a colored fingertip represented in the input image data.
Claim 14
Dependent Claim
34 words
An apparatus as claimed in claim 1 comprising image processing circuits for facial area and separate image processing circuits for colored fingertips, said circuits being connected to process the input image data in parallel.
Claim 15
Dependent Claim
44 words
An apparatus as claimed in claim 12 further comprising a result processor comprising means for receiving facial part and colored fingertip positional data to generate an output signal representing proximity of a colored fingertip to facial parts for assistance in communication of sign language.
Claim 16
Dependent Claim
18 words
An apparatus as claimed in claim 1 further comprising a video device for generating the image data stream.
Claim 17
Dependent Claim
20 words
An apparatus as claimed in claim 15, wherein the video device is a camera having an analog to digital converter.
Claim 18
Dependent Claim
53 words
An apparatus as claimed in claim 15 wherein the video device is a camera, and the camera is mounted on a frame adjacent a light source so that it is directed upwardly at an angle so that the face of a person seated next to the frame is within the field of view.
Claim 19
Independent Claim
129 words
A method of processing facial image data, the method comprising the steps of: receiving a digital color image data stream; monitoring color values in the image data to identify image data pixels representing facial parts, said monitoring step comprising sub-steps of carrying out three-dimensional color histogram matching in which there is backprojection with comparison of a generated three-dimensional color histogram with a template histogram generated off-line; determining states of the facial parts by monitoring positional coordinates of the identified pixels and using a counter to determine the area of the facial parts, said monitoring and determining steps being carried out by processing circuits interconnected for real time processing the image data in a pipelining manner; and outputting in real time a signal representing the determined facial part state data.
Claim 20
Independent Claim
129 words
A method of processing facial image data, the method comprising the steps of: receiving a digital color image data stream; monitoring color values in the image data to identify image data pixels representing facial parts, said monitoring step comprising sub-steps of carrying out three-dimensional color histogram matching in which there is backprojection with comparison of a generated three-dimensional color histogram with a template histogram generated off-line; determining states of the facial parts by monitoring positional coordinates of the identified pixels and using a counter to determine the area of the facial parts, said monitoring and determining steps being carried out by processing circuits interconnected for real time processing the image data in a pipelining manner; and outputting in real time a signal representing the determined facial part state data.
Claim 21
Dependent Claim
36 words
A method as claimed in claim 18, wherein the monitoring and determining steps are carried out to initially identify facial area and position, and subsequently to identify facial parts and determine states of the facial parts.
References Cited (Backward Citations)
US4484349A
Cited
US4975978A
Cited
US4991223A
Cited
US5253302A
Cited
US5459793A
Cited
EP0474307A2
Cited
US5291563A
Cited
US5473705A
Cited
US5467410A
Cited
US5343538A
Cited
US5495428A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2000-11-17 | 2001-05-19 | 2001-11-18 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2004-11-17 | 2005-05-19 | 2005-11-18 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2008-11-17 | 2009-05-19 | 2009-11-18 | $7700.00 | Window Closed / Lapsed |