US6970860B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Semi-automatic annotation of multimedia objects
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2000-10-30
•
Granted: 2005-11-29
•
Expires: 2020-10-30
Abstract
Abstract A multimedia object retrieval and annotation system integrates an annotation process with object retrieval and relevance feedback processes. The annotation process annotates multimedia objects, such as digital images, with semantically relevant keywords. The annotation process is performed in background, hidden from the user, as the user conducts normal searches. The annotation process is “semi-automatic” in that it utilizes both keyword-based information retrieval and content-based image retrieval techniques to automatically search for multimedia objects, and then encourages users to provide feedback on the retrieved objects. The user identifies objects as either relevant or irrelevant to the query keywords and based on this feedback, the system automatically annotates the objects with semantically relevant keywords and/or updates associations between the keywords and objects. As the retrieval-feedback-annotation cycle is repeated, the annotation coverage and accuracy of future searches continues to improve.
Inventors
- 👤 Wen-Yin Liu
- 👤 Hong-Jiang Zhang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 21:38
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/40; G06F16/48; G06F16/50; G06F16/58; G06F16/583; Y; Y10; Y10S; Y10S707/00; Y10S707/99931; Y10S707/99933; Y10S707/99934
Claims Analysis
Total Claims: 31
Claim 1
Independent Claim
188 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the first and second multimedia objects are relevant to the search query; maintaining associations between the keywords and the multimedia objects, the associations being weighted to indicate how relevant the keywords are to the multimedia objects; adjusting the weights of the associations based on the user's feedback, wherein the adjusting comprises increasing a weight of an association between the keyword and a particular multimedia object that is deemed relevant by the user or/and decreasing a weight of an association between the keyword and a particular multimedia object that is deemed irrelevant by the user; and annotating one or more of the second multimedia objects, which are deemed relevant by the user, with the keyword.
Claim 2
Independent Claim
188 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the first and second multimedia objects are relevant to the search query; maintaining associations between the keywords and the multimedia objects, the associations being weighted to indicate how relevant the keywords are to the multimedia objects; adjusting the weights of the associations based on the user's feedback, wherein the adjusting comprises increasing a weight of an association between the keyword and a particular multimedia object that is deemed relevant by the user or/and decreasing a weight of an association between the keyword and a particular multimedia object that is deemed irrelevant by the user; and annotating one or more of the second multimedia objects, which are deemed relevant by the user, with the keyword.
Claim 3
Dependent Claim
30 words
A method as recited in claim 1 , further comprising removing the keyword from the particular multimedia object in an event that the weight is less than a threshold value.
Claim 4
Dependent Claim
25 words
A method as recited in claim 1 , further comprising training how the first and second multimedia objects are identified based on the user's feedback.
Claim 5
Dependent Claim
36 words
A method as recited in claim 1 , further comprising refining the search to identify additional multimedia objects that contain content features similar to those of the multimedia objects indicated by the user as being relevant.
Claim 6
Dependent Claim
22 words
A method as recited in claim 1 , wherein the multimedia objects comprise one of digital images, video objects, and audio objects.
Claim 7
Dependent Claim
22 words
A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 1 .
Claim 8
Independent Claim
188 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the first and second multimedia objects are relevant to the search query; maintaining associations between the keywords and the multimedia objects, the associations being weighted to indicate how relevant the keywords are to the multimedia objects; adjusting the weights of the associations based on the user's feedback, wherein the adjusting comprises increasing a weight of an association between the keyword and a particular multimedia object that is deemed relevant by the user or/and decreasing a weight of an association between the keyword and a particular multimedia object that is deemed irrelevant by the user; and annotating one or more of the second multimedia objects, which are deemed relevant by the user, with the keyword.
Claim 9
Independent Claim
188 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the first and second multimedia objects are relevant to the search query; maintaining associations between the keywords and the multimedia objects, the associations being weighted to indicate how relevant the keywords are to the multimedia objects; adjusting the weights of the associations based on the user's feedback, wherein the adjusting comprises increasing a weight of an association between the keyword and a particular multimedia object that is deemed relevant by the user or/and decreasing a weight of an association between the keyword and a particular multimedia object that is deemed irrelevant by the user; and annotating one or more of the second multimedia objects, which are deemed relevant by the user, with the keyword.
Claim 10
Dependent Claim
21 words
A method as recited in claim 7 , wherein the retrieving comprises using content-based information retrieval to retrieve the multimedia objects.
Claim 11
Dependent Claim
26 words
A method as recited in claim 7 , wherein the retrieving comprises using both content-based information retrieval and semantic-based information retrieval to retrieve the multimedia objects.
Claim 12
Dependent Claim
21 words
A method as recited in claim 7 , wherein the monitoring comprises monitoring both feature-based relevance feedback and semantic-based relevance feedback.
Claim 13
Dependent Claim
16 words
A method as recited in claim 7 , wherein the annotating is hidden from the user.
Claim 14
Dependent Claim
44 words
A method as recited in claim 7 , wherein the annotating comprises: in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, weakening an association between the keyword and the particular multimedia object.
Claim 15
Dependent Claim
22 words
A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 7 .
Claim 16
Independent Claim
157 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the multimedia objects are relevant; annotating one or more of the multimedia objects based on the user's feedback, with a keyword; in an event that a particular multimedia object is deemed relevant by the user and not yet annotated with the keyword, adding the keyword to the particular multimedia object; and in an event that the particular multimedia object is deemed relevant by the user and is already annotated with the keyword, strengthening an association between the keyword and the particular multimedia object.
Claim 17
Independent Claim
157 words
A method comprising: identifying, in response to a search query, first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects, wherein the content features do not match the keyword in the search query; presenting the first and second multimedia objects to a user; monitoring feedback from the user as to which of the multimedia objects are relevant; annotating one or more of the multimedia objects based on the user's feedback, with a keyword; in an event that a particular multimedia object is deemed relevant by the user and not yet annotated with the keyword, adding the keyword to the particular multimedia object; and in an event that the particular multimedia object is deemed relevant by the user and is already annotated with the keyword, strengthening an association between the keyword and the particular multimedia object.
Claim 18
Dependent Claim
21 words
A method as recited in claim 14 , wherein the monitoring comprises monitoring both feature-based relevance feedback and semantic-based relevance feedback.
Claim 19
Dependent Claim
16 words
A method as recited in claim 14 , wherein the annotating is hidden from the user.
Claim 20
Dependent Claim
44 words
A method as recited in claim 14 , wherein the annotating comprises: in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, weakening an association between the keyword and the particular multimedia object.
Claim 21
Dependent Claim
41 words
A method as recited in claim 14 , wherein the annotating comprises: in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, removing the keyword from the particular multimedia object.
Claim 22
Dependent Claim
22 words
A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 14 .
Claim 23
Independent Claim
216 words
A system comprising: an information retrieval unit to retrieve multimedia objects from a database based on a search query first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects wherein the content features do not match the keyword in the search query; a relevance feedback unit to capture a user's feedback as to whether the multimedia objects are relevant to the search query; and an annotation unit to annotate, with the keyword, the second multimedia objects based on the user's feedback, wherein the search query comprises a keyword-based search query having at least one keyword and in an event that a particular multimedia object is deemed relevant by the user and is already annotated with the keyword, the annotation unit strengthens an association between the keyword and the particular multimedia object, and/or in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, removing the keyword from the particular multimedia object, and/or in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, weakening an association between the keyword and the particular multimedia object.
Claim 24
Independent Claim
216 words
A system comprising: an information retrieval unit to retrieve multimedia objects from a database based on a search query first multimedia objects having an associated keyword that matches a keyword in the search query, and identifying second multimedia objects that have content features similar to those of the first multimedia objects wherein the content features do not match the keyword in the search query; a relevance feedback unit to capture a user's feedback as to whether the multimedia objects are relevant to the search query; and an annotation unit to annotate, with the keyword, the second multimedia objects based on the user's feedback, wherein the search query comprises a keyword-based search query having at least one keyword and in an event that a particular multimedia object is deemed relevant by the user and is already annotated with the keyword, the annotation unit strengthens an association between the keyword and the particular multimedia object, and/or in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, removing the keyword from the particular multimedia object, and/or in an event that a particular multimedia object is deemed irrelevant by the user and is already annotated with the keyword, weakening an association between the keyword and the particular multimedia object.
Claim 25
Dependent Claim
22 words
A system as recited in claim 20 , wherein the search query comprises a keyword-based search query having one or more keywords.
Claim 26
Dependent Claim
23 words
A system as recited in claim 20 , wherein the search query comprises a content-based search query having one or more content features.
Claim 27
Dependent Claim
22 words
A system as recited in claim 20 , wherein the information retrieval unit employs both content-based information retrieval and semantic-based information retrieval.
Claim 28
Dependent Claim
85 words
A system as recited in claim 20 , wherein the information retrieval unit comprises: a query handler to handle both keyword-based queries having one or more search keywords and content-based queries having one or more content features of a multimedia object; and a feature and semantic matcher to identify at least one of (1) first multimedia objects having keywords that match the search keywords from a keyword-based query, and (2) second multimedia objects having content features similar to the content features of a content-based query.
Claim 29
Dependent Claim
22 words
A system as recited in claim 20 , wherein the relevance feedback unit employs both feature-based relevance feedback and semantic-based relevance feedback.
Claim 30
Dependent Claim
56 words
A system as recited in claim 20 , wherein: the search query comprises a keyword-based search query having at least one keyword; and in an event that a particular multimedia object is deemed relevant by the user and is not yet annotated with the keyword, the annotation unit adds the keyword to the particular multimedia object.
Claim 31
Dependent Claim
28 words
An image retrieval system as recited in claim 20 , wherein the relevance feedback unit comprises a feedback analyzer to train the system based on the user's feedback.
References Cited (Backward Citations)
US5579471A
Cited
US5572728A
Cited
US6314420B1
Cited
US5899999A
Cited
US6041335A
Cited
US6301586B1
Cited
US6324545B1
Cited
US6105055A
Cited
US6369811B1
Cited
US6574632B2
Cited
US6557042B1
Cited
US6408293B1
Cited
US6728406B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-11-28 | 2009-05-30 | 2009-11-29 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-11-28 | 2013-05-30 | 2013-11-29 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-11-28 | 2017-05-30 | 2017-11-29 | $7700.00 | Window Closed / Lapsed |