Patent US20050154716A1 successfully refreshed from Google Patents.
US20050154716A1
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for automated optimization of search result relevance
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2004-03-22
•
Granted: —
•
Expires: 2024-03-22
Abstract
Abstract A method, system, and computer-accessible medium are provided for automating the optimization of search result relevance in a search engine. The system and method continually collect data that represent various aspects of how a search result is performing and compare that performance data to the expected performance for the search result. The system and method further diagnose the possible causes of under performing results and automatically adjust the search engine operation to optimize the search result relevance.
Inventors
- 👤 Eric Watson
- 👤 Oliver Hurst-Hiller
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 12:40
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/30; G06F16/33; G06F16/332; G06F16/3325; G06F16/3326; G06F16/3331; G06F16/334; G06F16/80; G06F16/83; G06F16/835; G06F16/90; G06F16/95; G06F16/951; G06F16/953; G06F16/9535; G06F16/9538; G06F17/00; Y; Y10; Y10S; Y10S707/00; Y10S707/99931; Y10S707/99933; Y10S707/99935; Y10S707/99941; Y10S707/99942
Claims Analysis
Total Claims: 69
Claim 1
Independent Claim
111 words
1 . A method for automating the optimization of search results displayed in a search Web page, the method comprising: collecting data that represents a performance of a search result, the data originating from at least one of a plurality of sources of performance data; normalizing the collected data in accordance with a relative importance of the source of the data; comparing the normalized performance data to an expected performance data for the search result; diagnosing at least one possible cause for an under performing search result; and adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.
Claim 2
Dependent Claim
111 words
1 . A method for automating the optimization of search results displayed in a search Web page, the method comprising: collecting data that represents a performance of a search result, the data originating from at least one of a plurality of sources of performance data; normalizing the collected data in accordance with a relative importance of the source of the data; comparing the normalized performance data to an expected performance data for the search result; diagnosing at least one possible cause for an under performing search result; and adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.
Claim 3
Dependent Claim
45 words
2 . The method of claim 1 , wherein the at least one of the plurality of sources of performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.
Claim 4
Dependent Claim
54 words
3 . The method of claim 1 , wherein normalizing the collected data in accordance with a relative importance of the source of the data includes giving greater weight to the data from the more important sources and combining the data to reflect the relative importance of the source from which the data originated.
Claim 5
Dependent Claim
40 words
4 . The method of claim 2 , wherein comparing the normalized performance data to the expected performance data includes determining that the search result is under performing when the normalized performance is below or substantially below the expected performance.
Claim 6
Dependent Claim
36 words
5 . The method of claim 2 , wherein the most important source of data is implicit performance data, and normalizing the collected data includes giving implicit performance data substantially greater weight when combining the data.
Claim 7
Dependent Claim
23 words
6 . The method of claim 2 , wherein implicit performance data is automatically captured when a user interacts with the search result.
Claim 8
Dependent Claim
46 words
7 . The method of claim 6 , wherein implicit performance data includes at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.
Claim 9
Dependent Claim
34 words
8 . The method of claim 7 , wherein implicit performance data further identifies an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.
Claim 10
Dependent Claim
26 words
9 . The method of claim 6 , wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.
Claim 11
Dependent Claim
27 words
10 . The method of claim 6 , wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.
Claim 12
Dependent Claim
41 words
11 . The method of claim 2 , wherein explicit performance data includes data obtained from user response to a search engine operator-generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry.
Claim 13
Dependent Claim
23 words
12 . The method of claim 2 , wherein human-judged performance data includes data obtained from a human evaluation of the search result.
Claim 14
Dependent Claim
46 words
13 . The method of claim 2 , wherein relevance verification data includes data obtained from a verification test of the search result's relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.
Claim 15
Dependent Claim
32 words
14 . The method of claim 2 , wherein sample test data includes data obtained from a test of the search result's relevance performed on a sample of a subset of users.
Claim 16
Dependent Claim
83 words
15 . The method of claim 2 , wherein diagnosing at least one possible cause for an under performing search result includes considering at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.
Claim 17
Dependent Claim
61 words
16 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes modifying the search engine's search schema to change the search result generated for the search term, including at least one of reranking, removing, and replacing the search results.
Claim 18
Dependent Claim
74 words
17 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes modifying the search engine's search schema to augment a presentation of the search results generated for the search term, including at least one of highlighting, animating, enlarging, and repositioning an appearance of the search result on a search results web page.
Claim 19
Dependent Claim
39 words
18 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes increasing the search engine's spellchecker tolerance.
Claim 20
Dependent Claim
50 words
19 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes prompting the user to one of clarify or narrow the search term with an additional user input.
Claim 21
Dependent Claim
62 words
20 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes temporarily adjusting the operation of the search engine, and determining whether the adjustments have actually improved the search result performance before permanently adjusting the operation of the search engine.
Claim 22
Dependent Claim
43 words
21 . The method of claim 2 , wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes adjusting the operation of the search engine in real time.
Claim 23
Dependent Claim
45 words
22 . The method of claim 21 , wherein adjusting the operation of the search engine in real time includes intercepting the search result generated by the search engine and modifying the search result before the search engine displays the search result to the user.
Claim 24
Dependent Claim
127 words
23 . An automated search result optimization system comprising: an input data representing a performance of a search result generated by a search engine for a search term, the input data originating from a plurality of sources; a data collection process to collect the input performance data from the plurality of sources; a diagnostic process to compare a performance of the search result, as represented by the collected performance data, to an expected performance of the search result; and an adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine whenever the search result performance compares unfavorably to the expected performance, wherein the adjusted operation of the search engine is to improve the performance of the search result.
Claim 25
Dependent Claim
127 words
23 . An automated search result optimization system comprising: an input data representing a performance of a search result generated by a search engine for a search term, the input data originating from a plurality of sources; a data collection process to collect the input performance data from the plurality of sources; a diagnostic process to compare a performance of the search result, as represented by the collected performance data, to an expected performance of the search result; and an adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine whenever the search result performance compares unfavorably to the expected performance, wherein the adjusted operation of the search engine is to improve the performance of the search result.
Claim 26
Dependent Claim
46 words
24 . The system of claim 23 , wherein the at least one of the plurality of sources of input performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.
Claim 27
Dependent Claim
67 words
25 . The system of claim 23 , wherein the data collection process includes a normalizing process to normalize the collected data in accordance with a relative importance of the source of the data, and further wherein normalizing includes giving greater weight to the data from the more important sources and combining the data to reflect the relative importance of the source from which the data originated.
Claim 28
Dependent Claim
55 words
26 . The system of claim 25 , wherein the diagnostic process compares the performance of the search result as represented by the normalized collected performance data to an expected performance of the search result; and further, wherein the comparison is unfavorable when the performance is lower than or substantially lower than the expected performance.
Claim 29
Dependent Claim
36 words
27 . The system of claim 25 , wherein the most important source of data is implicit performance data, and normalizing the collected data includes giving implicit performance data substantially greater weight when combining the data.
Claim 30
Dependent Claim
23 words
28 . The system of claim 27 , wherein implicit performance data is automatically captured when a user interacts with the search result.
Claim 31
Dependent Claim
46 words
29 . The system of claim 28 , wherein implicit performance data includes at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.
Claim 32
Dependent Claim
39 words
30 . The system of claim 29 , wherein implicit performance data further includes data representing an identification of an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.
Claim 33
Dependent Claim
26 words
31 . The system of claim 28 , wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.
Claim 34
Dependent Claim
27 words
32 . The system of claim 28 , wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.
Claim 35
Dependent Claim
30 words
33 . The system of claim 24 , wherein explicit performance data includes data obtained from user response to a search engine operator-generated user satisfaction survey about the search result.
Claim 36
Dependent Claim
23 words
34 . The system of claim 24 , wherein human-judged performance data includes data obtained from a human evaluation of the search result.
Claim 37
Dependent Claim
46 words
35 . The system of claim 24 , wherein relevance verification data includes data obtained from a verification test of the search result's relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.
Claim 38
Dependent Claim
32 words
36 . The system of claim 24 , wherein sample test data includes data obtained from a test of the search result's relevance performed on a sample of a subset of users.
Claim 39
Dependent Claim
95 words
37 . The system of claim 24 , wherein the diagnostic process further includes diagnosing at least one possible reason why the performance of the search result compares unfavorably to the expected performance, including considering at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.
Claim 40
Dependent Claim
75 words
38 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to modify the search engine's search schema, wherein the modified search schema changes the way the search engine generates a search result for a search term, including at least one of reranking, removing, and replacing the search result.
Claim 41
Dependent Claim
85 words
39 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to modify the search engine's search schema, wherein the modified search schema augments the way the search engine presents the search result for a search term, including at least one of highlighting, animating, enlarging, and repositioning an appearance of the search result on a search result Web page.
Claim 42
Dependent Claim
44 words
40 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to increase the search engine's spellchecker tolerance.
Claim 43
Dependent Claim
55 words
41 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to prompt the user to one of clarify or narrow the search term with an additional user input.
Claim 44
Dependent Claim
74 words
42 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to temporarily adjust the operation of the search engine, and to further determine whether the temporary adjustment has actually improved the search result performance before generating an action to permanently adjust the operation of the search engine.
Claim 45
Dependent Claim
48 words
43 . The system of claim 23 , wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to adjust the operation of the search engine in real time.
Claim 46
Dependent Claim
53 words
44 . The system of claim 42 , wherein the action to adjust the operation of the search engine in real time includes an action to intercept the search result generated by the search engine and to further modify the search result before the search engine displays the search result to the user.
Claim 47
Dependent Claim
120 words
45 . One or more computer-accessible media having instructions for facilitating the automated optimization of a search result in a search result user interface, the instructions comprising: collecting data that represents a performance of a search result from at least one of a plurality of sources of performance data; normalizing the collected data in accordance with a relative importance of the source of the data; comparing the normalized performance data to an expected performance data for the search result; diagnosing at least one possible reason why the search result performance compares unfavorably to the expected performance; and adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.
Claim 48
Dependent Claim
120 words
45 . One or more computer-accessible media having instructions for facilitating the automated optimization of a search result in a search result user interface, the instructions comprising: collecting data that represents a performance of a search result from at least one of a plurality of sources of performance data; normalizing the collected data in accordance with a relative importance of the source of the data; comparing the normalized performance data to an expected performance data for the search result; diagnosing at least one possible reason why the search result performance compares unfavorably to the expected performance; and adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.
Claim 49
Dependent Claim
46 words
46 . The computer-accessible media of claim 45 , wherein the at least one of the plurality of sources of performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.
Claim 50
Dependent Claim
61 words
47 . The computer-accessible media of claim 45 , wherein the instruction to normalize the collected data in accordance with a relative importance of the source of the data includes instructions to give greater weight to the data from the more important sources and to combine the data to reflect the relative importance of the source from which the data originated.
Claim 51
Dependent Claim
47 words
48 . The computer-accessible media of claim 45 , wherein the instruction to compare the normalized performance data to the expected performance data includes an instruction to determine that the search result is under performing when the normalized performance is below or substantially below the expected performance.
Claim 52
Dependent Claim
43 words
49 . The computer-accessible media of claim 47 , wherein the most important source of data is implicit performance data, and the instruction to normalize the collected data includes an instruction to give implicit performance data substantially greater weight when combining the data.
Claim 53
Dependent Claim
31 words
50 . The computer-accessible media of claim 46 , wherein implicit performance data is data that is automatically captured by a search engine when a user interacts with a search result.
Claim 54
Dependent Claim
50 words
51 . The computer-accessible media of claim 50 , wherein implicit performance data includes data that represents at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.
Claim 55
Dependent Claim
35 words
52 . The computer-accessible media of claim 50 , wherein implicit performance data further identifies an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.
Claim 56
Dependent Claim
27 words
53 . The computer-accessible media of claim 50 , wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.
Claim 57
Dependent Claim
28 words
54 . The computer-accessible media of claim 50 , wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.
Claim 58
Dependent Claim
42 words
55 . The computer-accessible media of claim 46 , wherein explicit performance data includes data obtained from user response to a search engine operator-generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry.
Claim 59
Dependent Claim
24 words
56 . The computer-accessible media of claim 46 , wherein human-judged performance data includes data obtained from a human evaluation of the search result.
Claim 60
Dependent Claim
47 words
57 . The computer-accessible media of claim 46 , wherein relevance verification data includes data obtained from a verification test of the search result's relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.
Claim 61
Dependent Claim
33 words
58 . The computer-accessible media of claim 46 , wherein sample test data includes data obtained from a test of the search result's relevance performed on a sample of a subset of users.
Claim 62
Dependent Claim
95 words
59 . The computer-accessible media of claim 45 , wherein the instruction to diagnose at least one possible reason why the search result performance compares unfavorably to the expected performance includes an instruction to determine at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.
Claim 63
Dependent Claim
69 words
60 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to modify the search engine's search schema to change the search result generated for the search term, wherein to change the search result includes at least one of to rerank, remove, and replace the search result.
Claim 64
Dependent Claim
81 words
61 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to modify the search engine's search schema to augment a presentation of the search result generated for the search term, wherein to augment the presentation includes at least one of to highlight, animate, enlarge, and reposition a display of the search result on a search result Web page.
Claim 65
Dependent Claim
40 words
62 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to increase the search engine's spellchecker tolerance.
Claim 66
Dependent Claim
51 words
63 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to prompt the user to one of clarify or narrow the search term with an additional user input.
Claim 67
Dependent Claim
71 words
64 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to temporarily adjust the operation of the search engine, and an instruction to further determine whether the adjustments have actually improved the search result performance before performing an instruction to permanently adjust the operation of the search engine.
Claim 68
Dependent Claim
44 words
65 . The computer-accessible media of claim 45 , wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to adjust the operation of the search engine in real time.
Claim 69
Dependent Claim
56 words
66 . The computer-accessible media of claim 65 , wherein the instruction to adjust the operation of the search engine in real time includes an instruction to intercept the search result generated by the search engine and a further instruction to modify the search result before the search engine displays the search result to the user.
References Cited (Backward Citations)
US14331A
Cited
US6326962B1
Cited
US20020152204A1
Cited
US6223215B1
Cited
US6360227B1
Cited
US6269361B1
Cited
US20030033292A1
Cited
US6434550B1
Cited
US6529903B2
Cited
US20030172075A1
Cited
US6647383B1
Cited
US6526440B1
Cited
US6643639B2
Cited
US20050080771A1
Cited
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.