US6345111B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Multi-modal interface apparatus and method
Assignee: Toshiba Corp
•
Filing: 2000-06-13
•
Granted: 2002-02-05
•
Expires: 2020-06-13
Abstract
Abstract In the multi-modal interface apparatus of the present invention, a gaze object detection section always detects a user's gaze object. The user inputs at least one medium of sound information, character information, image information and operation information through a media input section. In order to effectively input and output information between the user and the apparatus, a personified image presentation section presents a personified image to the user based on the user's gaze object. A control section controls a reception of the inputted media from the media input section based on the user's gaze object.
Inventors
- 👤 Osamu Yamaguchi
- 👤 Kazuhiro Fukui
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:47
CPC Classifications
G; G06; G06F; G06F3/00; G06F3/01; G06F3/017; G06F3/011; G06F3/03; G06F3/033; G06F3/038; G06V; G06V40/00; G06V40/10; G06V40/16; H; H04; H04N; H04N21/00; H04N21/40; H04N21/41; H04N21/422; H04N21/42201; G06F2203/00; G06F2203/038; G06F2203/0381
Claims Analysis
Total Claims: 22
Claim 1
Independent Claim
176 words
Multi-modal interface apparatus, comprising: an image input unit configured to continually input an image of a user's entire face during the user's operation on a display; a face image processing unit configured to extract a feature from the user's face image, the first time that the user's operation of predetermined object on the display is detected a recognition decision unit configured to store the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object, and to recognize the user's face image newly inputted through said image input unit by referring to the dictionary pattern; and an object control unit configured to execute the event for the predetermined object on the display, when the user's face image newly inputted is recognized as the dictionary pattern, wherein said recognition decision unit deletes the dictionary pattern corresponding to the predetermined object, when said object control unit detects movement or deletion of the predetermined object on the display.
Claim 2
Dependent Claim
176 words
Multi-modal interface apparatus, comprising: an image input unit configured to continually input an image of a user's entire face during the user's operation on a display; a face image processing unit configured to extract a feature from the user's face image, the first time that the user's operation of predetermined object on the display is detected a recognition decision unit configured to store the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object, and to recognize the user's face image newly inputted through said image input unit by referring to the dictionary pattern; and an object control unit configured to execute the event for the predetermined object on the display, when the user's face image newly inputted is recognized as the dictionary pattern, wherein said recognition decision unit deletes the dictionary pattern corresponding to the predetermined object, when said object control unit detects movement or deletion of the predetermined object on the display.
Claim 3
Dependent Claim
47 words
The multi-modal interface apparatus according to claim 1 , wherein said recognition decision unit indicates to said face image processing unit to collect a plurality of the user's face images when said object control unit first detects the user's operation of the predetermined object on the display.
Claim 4
Dependent Claim
50 words
The multi-modal interface apparatus according to claim 2 , wherein said recognition decision unit indicates to said face image processing unit to extract the feature from the plurality of the user's face images when a number corresponding to the plurality of the user's face images is equal to predetermined number.
Claim 5
Dependent Claim
65 words
The multi-modal interface apparatus according to claim 1 , wherein said recognition decision unit recognizes an inputted user's face image by comparing the feature of the inputted user's face image with the dictionary pattern, when said object control unit detects a mouse event other than focusing the predetermined object on the display and if the dictionary pattern corresponding to the predetermined object is already registered.
Claim 6
Dependent Claim
51 words
The multi-modal interface apparatus according to claim 4 , wherein said recognition decision unit recognizes an inputted user's face image by comparing the feature of the inputted user's face image with the dictionary pattern, when said object control unit detects the user's operation other than the mouse event on the display.
Claim 7
Dependent Claim
46 words
The multi-modal interface apparatus according to claim 5 , wherein, said object control unit automatically executes the event corresponding to the dictionary pattern on the display, when said recognition decision unit determines that the feature of the inputted user's face image coincides with the dictionary pattern.
Claim 8
Dependent Claim
28 words
The multi-modal interface apparatus according to claim 1 , wherein the predetermined object is a window on the display, and wherein the event is focusing of the window.
Claim 9
Dependent Claim
37 words
The multi-modal interface apparatus according to claim 1 , wherein said recognition decision unit utilizes a subspace method in case of generation processing of the dictionary pattern and recognition processing of the newly inputted user's face image.
Claim 10
Dependent Claim
36 words
The multi-modal interface apparatus according to claim 1 , wherein the feature includes pixel values of the user's face area, and wherein a distribution of the pixel values represents the user's face direction and pupil position.
Claim 11
Dependent Claim
141 words
A multi-modal interface method, comprising the steps of: continually inputting an image of a user's entire face during the user's operation on a display; extracting a feature from the user's face image, the first time that the user's operation of a predetermined object on the display is detected; storing the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object; recognizing a newly inputted user's face image by referring to the dictionary pattern; executing the event for the predetermined object on the display, when the newly inputted user's face image is recognized as the dictionary pattern; and deleting the dictionary pattern corresponding to the predetermined object, when movement or deletion of the predetermined object on the display is detected.
Claim 12
Dependent Claim
141 words
A multi-modal interface method, comprising the steps of: continually inputting an image of a user's entire face during the user's operation on a display; extracting a feature from the user's face image, the first time that the user's operation of a predetermined object on the display is detected; storing the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object; recognizing a newly inputted user's face image by referring to the dictionary pattern; executing the event for the predetermined object on the display, when the newly inputted user's face image is recognized as the dictionary pattern; and deleting the dictionary pattern corresponding to the predetermined object, when movement or deletion of the predetermined object on the display is detected.
Claim 13
Dependent Claim
33 words
The multi-modal Interface method according to claim 10 , further comprising the step of: collecting a plurality of the user's face images, when the user's operation of the predetermined object is first detected.
Claim 14
Dependent Claim
43 words
The multi-modal interface method according to claim 11 , further comprising the step of: extracting the feature from the plurality of the user's face images, when a number corresponding to the plurality of the user's face images is equal to a predetermined number.
Claim 15
Dependent Claim
62 words
The multi-modal interface method according to claim 10 , further comprising the step of: recognizing an inputted user's face image by comparing the feature of the inputted user's face image with the dictionary pattern, when a mouse event other than focusing the predetermined object on the display is detected and if the dictionary pattern corresponding to the predetermined object is already registered.
Claim 16
Dependent Claim
48 words
The multi-modal interface method according to claim 13 , further comprising the step of: recognizing the inputted user's face image by comparing the feature of the inputted user's face image with the dictionary pattern, when the user's operation other than the mouse event on the display is detected.
Claim 17
Dependent Claim
40 words
The multi-modal interface method according to claim 14 , further comprising the step of: automatically executing the event corresponding to the dictionary pattern on the display, when the feature of the inputted user's face image coincides with the dictionary pattern.
Claim 18
Dependent Claim
28 words
The multi-modal interface method according to claim 10 , wherein the predetermined object is a window on the display, and wherein the event is focusing of the window.
Claim 19
Dependent Claim
31 words
The multi-modal interface method according to claim 10 , further comprising the step of: utilizing a subspace method when storing the dictionary pattern and recognizing the newly inputted user's face image.
Claim 20
Dependent Claim
36 words
The multi-modal interface method according to claim 10 , wherein the feature includes pixel values of the user's face area, and wherein a distribution of the pixel values represents the user's face direction and pupil position.
Claim 21
Dependent Claim
163 words
A computer readable memory containing computer readable instructions, comprising: an instruction unit to continually input an image of a user's entire face during the user's operation on a display; an instruction unit to extract a feature from the user's face image, the first time that the user's operation of predetermined object on the display is detected; an instruction unit to store the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object; an instruction unit to recognize a newly inputted user's face image by referring to the dictionary pattern; and an instruction unit to execute the event for the predetermined object on the display, when the newly inputted user's face image is recognized as the dictionary pattern; wherein said dictionary pattern corresponding to the predetermined object is deleted when movement or deletion of the predetermined object on the display is detected.
Claim 22
Dependent Claim
163 words
A computer readable memory containing computer readable instructions, comprising: an instruction unit to continually input an image of a user's entire face during the user's operation on a display; an instruction unit to extract a feature from the user's face image, the first time that the user's operation of predetermined object on the display is detected; an instruction unit to store the feature of the user's face image as a dictionary pattern and the user's operation selecting the predetermined object as an event, the dictionary pattern representing the user's gaze for the predetermined object; an instruction unit to recognize a newly inputted user's face image by referring to the dictionary pattern; and an instruction unit to execute the event for the predetermined object on the display, when the newly inputted user's face image is recognized as the dictionary pattern; wherein said dictionary pattern corresponding to the predetermined object is deleted when movement or deletion of the predetermined object on the display is detected.
References Cited (Backward Citations)
US4513317A
Cited
US4648052A
Cited
US4755045A
Cited
US5008946A
Cited
US5566271A
Cited
US5410609A
Cited
US5323470A
Cited
US5367454A
Cited
US6100857A
Cited
US5454043A
Cited
US5801704A
Cited
JPH08315118A
Cited
US5880731A
Cited
US5731805A
Cited
US5689619A
Cited
US5859642A
Cited
US6111517A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2005-02-04 | 2005-08-06 | 2006-02-05 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2009-02-04 | 2009-08-06 | 2010-02-05 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2013-02-04 | 2013-08-06 | 2014-02-05 | $7700.00 | Window Closed / Lapsed |