Patent US9101459B2 successfully refreshed from Google Patents.
US9101459B2
Active / In Force
US
⏱️ 7.1 years remaining
Apparatus and method for hierarchical object identification using a camera on glasses
Assignee: Orcam Technologies Ltd
•
Filing: 2013-12-20
•
Granted: 2015-08-11
•
Expires: 2033-12-20
Abstract
Abstract Apparatuses and a method are provided for providing environmental information to a user, who may be visually impaired. In one implementation, a method is provided for receiving real time image data including an object within the user environment. The method includes identifying a hand-related trigger in the image data. Upon identifying the hand-related trigger, the method includes executing a first search in the image data in an attempt to identify at least one object from a first category of objects. After initiating the first search, the method includes executing a second search in the image data in an attempt to identify at least one object from a second category of objects that differs from the first. The method also includes outputting audible feedback to the user associated with information related to an object or objects identified in the image data.
Inventors
- 👤 Yonatan Wexler
- 👤 Amnon Shashua
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:00
CPC Classifications
G; G09; G09B; G09B21/00; G09B21/001; G09B21/006; A; A61; A61F; A61F9/00; A61F9/08; G06F17/2765; G06; G06F; G06F3/00; G06F3/01; G06F3/011; G06F3/16; G06F40/00; G06F40/20; G06F40/279; G06K9/00221; G06K9/00288; G06K9/00671; G06K9/74; G06V; G06V10/00; G06V10/10; G06V10/17; G06V10/20; G06V10/22; G06V10/235; G06V10/24; G06V10/243; G06V10/25; G06V10/255; G06V20/00; G06V20/20; G06V20/60; G06V20/62; G06V30/00; G06V30/10; G06V30/14; G06V30/146; G06V30/1475; G06V30/1478; G06V30/40; G06V30/41; G06V30/414; G06V30/416; G06V30/418; G06V40/00; G06V40/10; G06V40/16; G06V40/172; G08; G08B; G08B3/00; G08B3/10; G08B6/00; G09B21/003; G10; G10L; G10L13/00; G10L13/043; H; H04; H04N; H04N23/00; H04N23/50; H04N23/51; H04N23/60; H04N23/61; H04N23/95; H04N23/951; H04N5/2251; H04N5/2252; H04N5/23229; H04N5/23232; G01; G01B; G01B11/00; G01B11/24; G02; G02C; G02C11/00; G02C11/10; G06V10/16; G06V30/144; G06V30/22; G06V30/226; G06V30/43; H04N23/66; H04N23/661
Claims Analysis
Total Claims: 29
Claim 1
Independent Claim
178 words
An apparatus for providing environmental information to a visually-impaired user, the apparatus comprising: a portable image sensor configured to be worn by the visually-impaired user and to capture real time image data from an environment of the visually-impaired user; and at least one portable processor device configured to; receive the image data and identify a hand-related trigger in the image data; after identifying the hand-related trigger, execute a first search in the image data to identify at least a first object from a first category of objects, wherein the first object is identified based on the hand-related trigger in the image data; after initiating the first search, execute a second search in at least one of the image data or results from the first search to identify at least a second object from a second category of objects, differing from the first category of objects; and cause audible feedback to be output to the visually-impaired user, wherein the audible feedback includes information related to at least one of the first and second objects identified in the image data.
Claim 2
Independent Claim
178 words
An apparatus for providing environmental information to a visually-impaired user, the apparatus comprising: a portable image sensor configured to be worn by the visually-impaired user and to capture real time image data from an environment of the visually-impaired user; and at least one portable processor device configured to; receive the image data and identify a hand-related trigger in the image data; after identifying the hand-related trigger, execute a first search in the image data to identify at least a first object from a first category of objects, wherein the first object is identified based on the hand-related trigger in the image data; after initiating the first search, execute a second search in at least one of the image data or results from the first search to identify at least a second object from a second category of objects, differing from the first category of objects; and cause audible feedback to be output to the visually-impaired user, wherein the audible feedback includes information related to at least one of the first and second objects identified in the image data.
Claim 3
Dependent Claim
34 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to execute the second search when no object from the first category is identified during the first search.
Claim 4
Dependent Claim
35 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to execute the second search only after an object from the first category is identified during the first search.
Claim 5
Dependent Claim
26 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to execute the first search and the second search sequentially.
Claim 6
Dependent Claim
50 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to: identify in the image data a particular context; and use the particular context to select the first category of objects and the second category of objects from a plurality of categories of objects.
Claim 7
Dependent Claim
33 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to repetitively execute the first search and the second search until at least one object is identified.
Claim 8
Dependent Claim
40 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to execute multiple category-based alternative actions, and after identifying an object from a particular category execute a category-based action associated with the particular category.
Claim 9
Dependent Claim
34 words
The apparatus of claim 1 , wherein the first search includes searching for objects that prompt audible warnings of risks and the second search includes searching for objects that prompt audible identification of objects.
Claim 10
Dependent Claim
39 words
The apparatus of claim 1 , wherein the at least one processor device is further configured to execute at least one additional search in the image data to identify an object from at least one other category of objects.
Claim 11
Dependent Claim
54 words
The apparatus of claim 9 , wherein the at least one processor device is further configured to cause hierarchical searching based on categories, and wherein the categories include at least three of the following: spatial guidance, warning of risks, objects to be identified, text to be read, scene identification, and text in the wild.
Claim 12
Dependent Claim
32 words
The apparatus of claim 1 , wherein the first category of objects includes at least one of: a bus, a taxi, a train, a traffic light, a sidewalk, and a pedestrian crossing.
Claim 13
Dependent Claim
21 words
The apparatus of claim 11 , wherein the second category of objects includes a landscape previously recorded by the visually-impaired user.
Claim 14
Dependent Claim
34 words
The apparatus of claim 12 , wherein the at least one processor device is further configured to identify a third object from a third category of objects, wherein the third category includes branded products.
Claim 15
Dependent Claim
33 words
The apparatus of claim 12 , wherein the at least one processor device is further configured to identify a third object from a third category of objects, wherein the third category includes text.
Claim 16
Independent Claim
246 words
An apparatus for providing environmental information to a user, the apparatus comprising: an image sensor configured to be worn by the user and to capture real time image data from an environment of the user; and at least one processor device configured to; receive the image data from the image sensor; execute, after identifying a hand-related trigger in the image data, a first search in the image data in an attempt to identify a first object from a first category of objects wherein the first object is identified based on the hand-related trigger in the image data; execute a second search in the image data in an attempt to identify a second object from a second category of objects, differing from the first category of objects; execute a third search in the image data in an attempt to identify a third object from a third category of objects, differing from the first category of objects and the second category of objects; execute a fourth search in the image data in an attempt to identify a fourth object from a fourth category of objects, differing from the first category of objects, the second category of objects, and the third category of objects; execute an action associated with at least one of: the first object, the second object, the third object, and the fourth object identified in the image data; and cause audible feedback to be output, wherein the audible feedback includes information related to an identified object.
Claim 17
Independent Claim
246 words
An apparatus for providing environmental information to a user, the apparatus comprising: an image sensor configured to be worn by the user and to capture real time image data from an environment of the user; and at least one processor device configured to; receive the image data from the image sensor; execute, after identifying a hand-related trigger in the image data, a first search in the image data in an attempt to identify a first object from a first category of objects wherein the first object is identified based on the hand-related trigger in the image data; execute a second search in the image data in an attempt to identify a second object from a second category of objects, differing from the first category of objects; execute a third search in the image data in an attempt to identify a third object from a third category of objects, differing from the first category of objects and the second category of objects; execute a fourth search in the image data in an attempt to identify a fourth object from a fourth category of objects, differing from the first category of objects, the second category of objects, and the third category of objects; execute an action associated with at least one of: the first object, the second object, the third object, and the fourth object identified in the image data; and cause audible feedback to be output, wherein the audible feedback includes information related to an identified object.
Claim 18
Dependent Claim
25 words
The apparatus of claim 15 , wherein the at least one processor device is further configured to identify the hand-related trigger in the image data.
Claim 19
Dependent Claim
37 words
The apparatus of claim 15 , wherein the apparatus further comprises a user interface and the at least one processor device is further configured to identify the hand-related trigger based on input received via the user interface.
Claim 20
Dependent Claim
26 words
The apparatus of claim 15 , wherein the third search is initiated after the second search and the second search is initiated after the first search.
Claim 21
Dependent Claim
71 words
The apparatus of claim 15 , wherein the at least one processor device is further configured to execute the second search when no object from the first category is identified during the first search, to execute the third search when no object from the second category is identified during the second search, and to execute the fourth search when no object from the third category is identified during the third search.
Claim 22
Dependent Claim
35 words
The apparatus of claim 15 , wherein the at least one processor device is further configured to execute the second search only after an object from the first category is identified during the first search.
Claim 23
Independent Claim
122 words
A method for providing environmental information to a visually-impaired user, the method comprising: identifying a hand-related trigger in captured image data; after identifying the hand-related trigger, executing a first search in the image data in an attempt to identify at least one object from a first category of objects, wherein the first object is identified based on the hand-related trigger in the image data; after initiating the first search, executing a second search in the image data in an attempt to identify at least one object from a second category of objects, differing from the first category of objects; and outputting audible feedback to the visually-impaired user, wherein the audible feedback includes information related to an object identified in the image data.
Claim 24
Independent Claim
122 words
A method for providing environmental information to a visually-impaired user, the method comprising: identifying a hand-related trigger in captured image data; after identifying the hand-related trigger, executing a first search in the image data in an attempt to identify at least one object from a first category of objects, wherein the first object is identified based on the hand-related trigger in the image data; after initiating the first search, executing a second search in the image data in an attempt to identify at least one object from a second category of objects, differing from the first category of objects; and outputting audible feedback to the visually-impaired user, wherein the audible feedback includes information related to an object identified in the image data.
Claim 25
Dependent Claim
26 words
A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 21 .
Claim 26
Independent Claim
199 words
An apparatus for providing environmental information to a visually-impaired user, the apparatus comprising: a portable image sensor configured to be worn by the visually-impaired user and to capture real time image data from an environment of the visually-impaired user; and at least one portable processor device configured to: receive the image data and identify a hand-related trigger in the image data; after identifying the hand-related trigger, identify at least one object in the image data based on the hand-related trigger in the image data; determine first information related to the at least one object identified in the image data; cause first audible feedback to be output to the visually-impaired user, wherein the first audible feedback comprises the first information related to the at least one object identified in the image data; determine, after identifying that the hand-related trigger remains associated with the at least one object for a predetermined period of time, second information related to the at least one object identified in the image data; and cause second audible feedback to be output to the visually-impaired user, wherein the second audible feedback includes the second information related to at least the first object identified in the image data.
Claim 27
Independent Claim
199 words
An apparatus for providing environmental information to a visually-impaired user, the apparatus comprising: a portable image sensor configured to be worn by the visually-impaired user and to capture real time image data from an environment of the visually-impaired user; and at least one portable processor device configured to: receive the image data and identify a hand-related trigger in the image data; after identifying the hand-related trigger, identify at least one object in the image data based on the hand-related trigger in the image data; determine first information related to the at least one object identified in the image data; cause first audible feedback to be output to the visually-impaired user, wherein the first audible feedback comprises the first information related to the at least one object identified in the image data; determine, after identifying that the hand-related trigger remains associated with the at least one object for a predetermined period of time, second information related to the at least one object identified in the image data; and cause second audible feedback to be output to the visually-impaired user, wherein the second audible feedback includes the second information related to at least the first object identified in the image data.
Claim 28
Dependent Claim
100 words
The apparatus of claim 23 , wherein the at least one processor device is further configured to: identify in the image data a particular context; use the particular context to determine information about at least the first object identified in the image data that is of the highest priority; determine the first information based at least on the highest priority information; use the particular context to determine information about at least the first object identified in the image data that is of the next highest priority; and determine the second information based at least on the next highest priority information.
Claim 29
Dependent Claim
36 words
The apparatus of claim 23 , wherein the at least one processor device is further configured to identify at least one additional object in the image data based on the hand-related trigger in the image data.
References Cited (Backward Citations)
US6115482A
Cited
US20050208457A1
Cited
US20060017810A1
Cited
EP2065871A1
Cited
EP2490155A1
Cited
US20120212593A1
Cited
US20130169536A1
Cited
US20130271584A1
Cited
US20120330646A1
Cited
US20130035742A1
Cited
US20130107026A1
Cited
US20130250078A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2018-08-10 | 2019-02-09 | 2019-08-11 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2022-08-10 | 2023-02-09 | 2023-08-11 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2026-08-10 | 2027-02-09 | 2027-08-11 | $7700.00 | Open for Payment |