US9436707B2
Active / In Force
US
⏱️ 7.7 years remaining
Content-based image ranking
Assignee: Google LLC
•
Filing: 2014-07-14
•
Granted: 2016-09-06
•
Expires: 2034-07-14
Abstract
Abstract Methods, systems, and apparatus, including computer program products, for ranking search results for queries. The method includes calculating a visual similarity score for one or more pairs of images in a plurality of images based on visual features of images in each of the one or more pairs; building a graph of images by linking each of one or more images in the plurality of images to one or more nearest neighbor images based on the visual similarity scores; associating a respective score with each of one or more images in the graph based on data indicative of user behavior relative to the image as a search result for a query; and determining a new score for each of one or more images in the graph based on the respective score of the image, and the respective scores of one or more nearest neighbors to the image.
Inventors
- 👤 Sanjiv Kumar
- 👤 Henry Allan Rowley
- 👤 Ameesh Makadia
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 09:39
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/50; G06F16/54; G06F17/30274; G06F16/58; G06F16/583; G06F17/30247; G06F18/00; G06F18/20; G06F18/22; G06F18/23; G06F18/232; G06F18/2323; G06K9/6215; G06K9/6224; G06V; G06V10/00; G06V10/70; G06V10/74; G06V10/761; G06K2209/27; G06V2201/00; G06V2201/10
Claims Analysis
Total Claims: 20
Claim 1
Independent Claim
229 words
A computer-implemented method comprising: extracting a first set of features from visual content of a particular image; accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 2
Dependent Claim
229 words
A computer-implemented method comprising: extracting a first set of features from visual content of a particular image; accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 3
Dependent Claim
19 words
The method of claim 1 , further comprising ranking each of the plurality of images based on the score.
Claim 4
Dependent Claim
23 words
The method of claim 1 , further comprising determining the distance measure based on a combination of basic distances from multiple visual features.
Claim 5
Dependent Claim
18 words
The method of claim 1 , further comprising determining the distance measure based on a joint equal contribution.
Claim 6
Dependent Claim
51 words
The method of claim 1 , wherein determining the score comprises calculating C=(1−α*(I−(α*W)) −1 *c, where c is a vector representing the visual similarity scores, C is a vector representing the determined scores, W is a matrix representing normalized weights, I is an identity matrix, and α is a determined parameter.
Claim 7
Dependent Claim
17 words
The method of claim 5 , where determining the score includes: iteratively calculating the score until convergence.
Claim 8
Dependent Claim
245 words
A non-transitory computer readable medium encoded with a computer program that causes one or more processors to perform operations comprising: accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; extracting a first set of features from visual content of a particular image; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 9
Dependent Claim
245 words
A non-transitory computer readable medium encoded with a computer program that causes one or more processors to perform operations comprising: accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; extracting a first set of features from visual content of a particular image; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 10
Dependent Claim
35 words
The non-transitory computer readable medium of claim 7 , wherein the computer program causes the one or more processors to perform operations further comprising ranking each of the plurality of images based on the score.
Claim 11
Dependent Claim
39 words
The non-transitory computer readable medium of claim 7 , wherein the computer program causes the one or more processors to perform operations further comprising determining the distance measure based on a combination of basic distances from multiple visual features.
Claim 12
Dependent Claim
34 words
The non-transitory computer readable medium of claim 7 , wherein the computer program causes the one or more processors to perform operations further comprising determining the distance measure based on a joint equal contribution.
Claim 13
Dependent Claim
52 words
The non-transitory computer readable medium of claim 9 , wherein determining the score comprises calculating C=(1−.alpha.)*(I−(.alpha.*W)).sup.−1*c, where c is a vector representing the visual similarity scores, C is a vector representing the determined scores, W is a matrix representing normalized weights, I is an identity matrix, and .alpha. is a determined parameter.
Claim 14
Independent Claim
260 words
A system, comprising: a machine readable storage device including a computer program; and one or more processors that interact with the machine readable storage device and upon execution of the computer program performs operations comprising: accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; extracting a first set of features from visual content of a particular image; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 15
Independent Claim
260 words
A system, comprising: a machine readable storage device including a computer program; and one or more processors that interact with the machine readable storage device and upon execution of the computer program performs operations comprising: accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; extracting a first set of features from visual content of a particular image; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.
Claim 16
Dependent Claim
32 words
The system of claim 12 , wherein the computer program causes the one or more processors to perform operations further comprising ranking each of the plurality of images based on the score.
Claim 17
Dependent Claim
36 words
The system of claim 12 , wherein the computer program causes the one or more processors to perform operations further comprising determining the distance measure based on a combination of basic distances from multiple visual features.
Claim 18
Dependent Claim
19 words
The system of claim 12 , wherein further comprising determining the distance measure based on a joint equal contribution.
Claim 19
Dependent Claim
51 words
The system of claim 12 , wherein determining the score comprises calculating C=(1−α)*(I−(α*W)) −1 *c, where c is a vector representing the visual similarity scores, C is a vector representing the determined scores, W is a matrix representing normalized weights, I is an identity matrix, and α is a determined parameter.
Claim 20
Dependent Claim
17 words
The system of claim 16 , wherein determining the score includes: iteratively calculating the score until convergence.
References Cited (Backward Citations)
US20020188602A1
Cited
EP1696353B1
Cited
JP2007207031A
Cited
US20070271226A1
Cited
WO2007136861A2
Cited
WO2007136861A3
Cited
KR20070115194A
Cited
US20070296993A1
Cited
US20080021928A1
Cited
US8422832B2
Cited
US20130198601A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2019-09-06 | 2020-03-07 | 2020-09-06 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2023-09-06 | 2024-03-07 | 2024-09-06 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2027-09-06 | 2028-03-07 | 2028-09-06 | $7700.00 | Pending Window |