US8194939B2
Active / In Force
US
⏱️ 3.7 years remaining
Using relevance feedback in face recognition
Assignee: Oxford University Innovation Ltd
•
Filing: 2010-08-03
•
Granted: 2012-06-05
•
Expires: 2030-08-03
Abstract
Abstract Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.
Inventors
- 👤 Keren O. Perlmutter
- 👤 Sharon M. Perlmutter
- 👤 Joshua Alspector
- 👤 Mark Everingham
- 👤 Alex Holub
- 👤 Andrew Zisserman
- 👤 Pietro Perona
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:45
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/50; G06F16/58; G06F16/583; G06F16/5838; G06F16/5854; G06F18/00; G06F18/20; G06F18/21; G06F18/2163; G06F18/217; G06F18/2178; G06V; G06V10/00; G06V10/70; G06V10/77; G06V10/778; G06V10/7784; G06V40/00; G06V40/10; G06V40/16; G06V40/168; G06V40/171; G06V40/172; G11; G11B; G11B27/00; G11B27/10; G11B27/102; G11B27/105; G11B27/19; G11B27/28
Claims Analysis
Total Claims: 50
Claim 1
Independent Claim
119 words
A computer-implemented method for recognizing faces within images, the method comprising: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be printed; determining, by at least one processor, that a face in the printed search result image matches the query face; and updating the set of search result images based on the determination that the face in the printed search result image matches the query face.
Claim 2
Dependent Claim
119 words
A computer-implemented method for recognizing faces within images, the method comprising: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be printed; determining, by at least one processor, that a face in the printed search result image matches the query face; and updating the set of search result images based on the determination that the face in the printed search result image matches the query face.
Claim 3
Dependent Claim
97 words
The method of claim 1 wherein determining a set of search result images includes: creating a query feature vector for the query face; forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; determining distances between the query feature vector and the search feature vectors; selecting one or more search feature vectors that are within a selected distance of the query feature vector; and designating images that include faces that correspond to the selected search feature vectors as the set of search result images.
Claim 4
Dependent Claim
55 words
The method of claim 2 wherein forming a set of search feature vectors includes: detecting a class for the query face; determining faces in a set of faces that have the same class as the query face; and forming the set of search feature vectors by selecting feature vectors that correspond to the determined faces.
Claim 5
Dependent Claim
36 words
The method of claim 2 wherein determining distances between the query feature vector and the search feature vectors comprises determining a weighted Euclidean distance between the query feature vector and each of the search feature vectors.
Claim 6
Dependent Claim
58 words
The method-of claim 2 wherein creating the query feature vector comprises: locating the query face within a query image; detecting facial features of the query face; warping the query face based on the detected facial features; extracting information regarding the detected facial features from the warped face; and forming the query feature vector based on the extracted information.
Claim 7
Dependent Claim
52 words
The method of claim 5 further comprising: detecting non-face based features from the query image; extracting information regarding the non-face based features from the query image; and forming the query feature vector based on the extracted information regarding the detected non-face based features and the extracted information regarding the detected facial features.
Claim 8
Dependent Claim
65 words
The method of claim 2 wherein: accessing at least one query face comprises accessing multiple query faces; creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors.
Claim 9
Dependent Claim
104 words
The method of claim 2 further comprising: identifying negative faces that are determined to not be the same as the query face; and creating negative feature vectors for the negative faces, wherein: accessing at least one query face comprises accessing multiple query faces; creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors and determining distances between the negative feature vectors and the search feature vectors.
Claim 10
Dependent Claim
66 words
The method of claim 1 , further comprising: inferring additional feedback based on the determination that the face in the printed search result images matches the query face, wherein updating the set of search result images includes updating the set of search result images based on the determination that the face in the printed search result images matches the query face and the inferred additional feedback.
Claim 11
Dependent Claim
28 words
The method of claim 9 wherein the inferred additional feedback includes information about faces in the search result images other than the face in the printed search result.
Claim 12
Dependent Claim
35 words
The method of claim 9 wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.
Claim 13
Dependent Claim
80 words
The method of claim 1 further comprising: accessing a set of images that include faces that have not been labeled; sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.
Claim 14
Dependent Claim
17 words
The method of claim 1 wherein the search result images comprise images within an online photograph album.
Claim 15
Dependent Claim
41 words
The method of claim 13 wherein the indication of the search result images provided to the user is an indication that the search result images have been associated with the query face, wherein the query face is in a labeled image.
Claim 16
Dependent Claim
22 words
The method of claim 13 wherein the indication of the search result images provided to the user includes the search result images.
Claim 17
Dependent Claim
16 words
The method of claim 1 wherein the search result images comprise images within a social network.
Claim 18
Independent Claim
222 words
A system comprising: one or more processors; and a memory device that stores instructions, the instructions being executable by the one or more processors to cause the one or more processors to implement: a feature extractor configured to: access at least one query face, create a query feature vector for the query face, and form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; a face classifier configured to: determine distances between the query feature vector and the search feature vectors, select one or more search feature vectors that are within a selected distance of the query feature vector, and designate images that include faces that correspond to the selected search feature vectors as a set of search result images; an interface manager configured to: provide an indication of the search result images to a user, and receive an indication that the user has caused at least one of the provided search result images to be printed; and a feedback manager configured to determine that a face in the printed search result image matches the query face, wherein the face classifier is also configured to, update the set of search result images based on the determination that the face in the printed search result image matches the query face.
Claim 19
Independent Claim
222 words
A system comprising: one or more processors; and a memory device that stores instructions, the instructions being executable by the one or more processors to cause the one or more processors to implement: a feature extractor configured to: access at least one query face, create a query feature vector for the query face, and form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; a face classifier configured to: determine distances between the query feature vector and the search feature vectors, select one or more search feature vectors that are within a selected distance of the query feature vector, and designate images that include faces that correspond to the selected search feature vectors as a set of search result images; an interface manager configured to: provide an indication of the search result images to a user, and receive an indication that the user has caused at least one of the provided search result images to be printed; and a feedback manager configured to determine that a face in the printed search result image matches the query face, wherein the face classifier is also configured to, update the set of search result images based on the determination that the face in the printed search result image matches the query face.
Claim 20
Dependent Claim
40 words
The method of claim 9 , wherein inferring additional feedback further includes inferring extra information for the other faces within one or more of the provided search result images for which the user has not provided explicit or implicit feedback.
Claim 21
Dependent Claim
114 words
A computer-implemented method for recognizing faces within images, the method comprising: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 22
Dependent Claim
114 words
A computer-implemented method for recognizing faces within images, the method comprising: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 23
Dependent Claim
97 words
The method of claim 19 wherein determining a set of search result images includes: creating a query feature vector for the query face; forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; determining distances between the query feature vector and the search feature vectors; selecting one or more search feature vectors that are within a selected distance of the query feature vector; and designating images that include faces that correspond to the selected search feature vectors as the set of search result images.
Claim 24
Dependent Claim
66 words
The method of claim 19 , further comprising: inferring additional feedback based on the determination that the face in the downloaded search result images matches the query face, wherein updating the set of search result images includes updating the set of search result images based on the determination that the face in the downloaded search result images matches the query face and the inferred additional feedback.
Claim 25
Dependent Claim
40 words
The method of claim 21 , wherein inferring additional feedback further includes inferring extra information for the other faces within one or more of the provided search result images for which the user has not provided explicit or implicit feedback.
Claim 26
Dependent Claim
28 words
The method of claim 21 wherein the inferred additional feedback includes information about faces in the search result images other than the face in the downloaded search result.
Claim 27
Dependent Claim
35 words
The method of claim 21 wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.
Claim 28
Dependent Claim
80 words
The method of claim 19 further comprising: accessing a set of images that include faces that have not been labeled; sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.
Claim 29
Dependent Claim
17 words
The method of claim 19 wherein the search result images comprise images within an online photograph album.
Claim 30
Dependent Claim
16 words
The method of claim 19 wherein the search result images comprise images within a social network.
Claim 31
Independent Claim
135 words
A system for recognizing faces within images, comprising: a processor; and a memory, wherein the memory stores instructions that, when executed by the processor, cause the processor to perform the operations of: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 32
Independent Claim
135 words
A system for recognizing faces within images, comprising: a processor; and a memory, wherein the memory stores instructions that, when executed by the processor, cause the processor to perform the operations of: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 33
Dependent Claim
107 words
The system of claim 28 wherein the memory stores instructions that, when executed by the processor, cause the processor to perform the operations of: creating a query feature vector for the query face; forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; determining distances between the query feature vector and the search feature vectors; selecting one or more search feature vectors that are within a selected distance of the query feature vector; and designating images that include faces that correspond to the selected search feature vectors as the set of search result images.
Claim 34
Dependent Claim
83 words
The system of claim 29 wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform the operations of: inferring additional feedback based on the determination that the face in the downloaded search result images matches the query face, wherein updating the set of search result images includes updating the set of search result images based on the determination that the face in the downloaded search result images matches the query face and the inferred additional feedback.
Claim 35
Dependent Claim
40 words
The system of claim 30 , wherein inferring additional feedback further includes inferring extra information for the other faces within one or more of the provided search result images for which the user has not provided explicit or implicit feedback.
Claim 36
Dependent Claim
29 words
The system of claim 30 , wherein the inferred additional feedback includes information about faces in the search result images other than the face in the downloaded search result.
Claim 37
Dependent Claim
36 words
The system of claim 30 , wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.
Claim 38
Dependent Claim
98 words
The system of claim 28 wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform the operations of: accessing a set of images that include faces that have not been labeled; sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.
Claim 39
Dependent Claim
18 words
The system of claim 28 , wherein the search result images comprise images within an online photograph album.
Claim 40
Dependent Claim
16 words
The system of claim 28 wherein the search result images comprise images within a social network.
Claim 41
Dependent Claim
130 words
A computer-readable medium having stored thereon executable instructions that, when executed by at least one processor, cause the at least one processor to perform the following operations: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 42
Dependent Claim
130 words
A computer-readable medium having stored thereon executable instructions that, when executed by at least one processor, cause the at least one processor to perform the following operations: accessing at least one query face; determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face; providing an indication of the search result images to a user; receiving an indication that the user has caused at least one of the provided search result images to be downloaded; determining that a face in the downloaded search result image matches the query face; and updating the set of search result images based on the determination that the face in the downloaded search result image matches the query face.
Claim 43
Dependent Claim
113 words
The computer-readable medium of claim 37 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of: creating a query feature vector for the query face; forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces; determining distances between the query feature vector and the search feature vectors; selecting one or more search feature vectors that are within a selected distance of the query feature vector; and designating images that include faces that correspond to the selected search feature vectors as the set of search result images.
Claim 44
Dependent Claim
88 words
The computer-readable medium of claim 37 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of: inferring additional feedback based on the determination that the face in the downloaded search result images matches the query face, wherein updating the set of search result images includes updating the set of search result images based on the determination that the face in the downloaded search result images matches the query face and the inferred additional feedback.
Claim 45
Dependent Claim
41 words
The computer-readable medium of claim 39 , wherein inferring additional feedback further includes inferring extra information for the other faces within one or more of the provided search result images for which the user has not provided explicit or implicit feedback.
Claim 46
Dependent Claim
30 words
The computer-readable medium of claim 39 , wherein the inferred additional feedback includes information about faces in the search result images other than the face in the downloaded search result.
Claim 47
Dependent Claim
37 words
The computer-readable medium of claim 39 , wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.
Claim 48
Dependent Claim
103 words
The computer-readable medium of claim 37 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of: accessing a set of images that include faces that have not been labeled; sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.
Claim 49
Dependent Claim
19 words
The computer-readable medium of claim 37 , wherein the search result images comprise images within an online photograph album.
Claim 50
Dependent Claim
18 words
The computer-readable medium of claim 37 , wherein the search result images comprise images within a social network.
References Cited (Backward Citations)
US5642431A
Cited
US5850470A
Cited
US5893095A
Cited
US6222939B1
Cited
US6563950B1
Cited
US6480840B2
Cited
US6574632B2
Cited
US6832006B2
Cited
US7106903B2
Cited
US20040264780A1
Cited
US20050084154A1
Cited
US20050226509A1
Cited
US20060251292A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2015-06-05 | 2015-12-05 | 2016-06-05 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2019-06-05 | 2019-12-05 | 2020-06-05 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2023-06-05 | 2023-12-05 | 2024-06-05 | $7700.00 | Window Closed / Lapsed |