Generalized term frequency scores in information retrieval systems

Claim 3

The method of claim 1 , wherein the documents are Web pages.

Claim 4

The method of claim 1 , wherein the documents are Web sites.

Claim 5

The method of claim 1 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms used in a search query.

Claim 6

The method of claim 1 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms used in a search query, with preselected stop terms eliminated.

Claim 7

The method of claim 1 wherein additional terms are chosen to be used in selecting documents which may be of interest based upon the terms used in the search query.

Claim 8

The method of claim 1 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms in a document identified as having characteristics that are desired in the documents to be selected.

Claim 9

The method of claim 1 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms in a document identified as having characteristics that are desired in the documents to be selected, with preselected stop terms eliminated.

Claim 10

The method of claim 1 , wherein additional terms are chosen to be used in selecting documents based upon the terms which occur in the document identified as having characteristics that are desired in the documents to be selected.

Claim 11

The method of claim 1 , wherein the terms chosen to be used in selecting documents comprise the terms in a document being accessed by a user.

Claim 12

The method of claim 1 , wherein the terms chosen to be used in selecting documents comprise the terms in a document being accessed by a user, with preselected stop terms eliminated.

Claim 13

The method of claim 1 , wherein additional terms are chosen to be used in selecting documents based upon the terms which occur in the document being accessed by the user.

Claim 14

The method of claim 1 , wherein the documents in the collection are ranked in order of their generalized term frequency scores S D , and a predetermined number N of documents with the highest scores is selected.

Claim 15

The method of claim 1 , wherein all documents with generalized term frequency scores S D above a predetermined cutoff score S are selected.

Claim 16

The method of claim 1 , wherein, for the plurality of documents D in the collection, a weight W SD is assigned to each segment S i of the document, and the generalized term frequency score S D for a document D is based upon Robertson's term frequency score, such that: S D = ∑ T = 1 T 0   ∑ S i = 1 S 0   TF STD * IDF ST where: S D is the total score for the document D, T 0 is the number of terms which occur in the collection of terms included in the search, S 0 is the number of segments in the document D, TF STD =Robertson's generalized term frequency for Term T in Segment S i of Document D = G STD /( G STD +K 1 +K 2 *W SD *( H SD /H SO )), where: G STD =the generalized term count for Term T in segment S i of Document D, = W SD *W STD *N STD , where: W SD is the weight assigned to segment S i of document D, W STD is the weight assigned to term T in segment S i of document D, and N STD is the number of times the term T occurs in segment S i of document D, H SD =the generalized length of segment S i of document D, H SD = ∑ T = 1 L SD  W STD * N STD where: L SD is the number of different terms in segment S i of document D, H SO =the generalized average length of segment S i of documents in the collection being searched, H SO = ∑ N = 1 N 0  ( W SD * H SD ) / ∑ N = 1 N 0  W SD , where: N 0 is the number of documents in the collection, and K 1 and K 2 are constants, IDF ST =the generalized inverted document frequency for term T, IDF ST =log(( N 0 +K 3 )/ N ST )/log( N 0 +K 4 ) where: N 0 is the number of documents in the collection N ST is the number of documents in the collection containing the term T in the segment S i , K 3 and K 4 are constants.

Claim 17

The method of claim 15 , wherein K 1 =0.5, K 2 =1.5, K 3 =0.5, and K 4 =1.0.

Claim 18

The method of claim 15 , wherein the weights W iD assigned to the i th segment of the documents in the collection are equal.

Claim 19

The method of claim 17 , wherein the weights W SD assigned to the segments of a document D in the collection have the property that ∑ S i = 1 S 0   W SD = 1.

Claim 20

The method of claim 17 , wherein the weights W iD are selected specifically for the collection of documents from which documents are to be chosen by carrying out test searches with different weights, and selecting for use the weights which yield the most useful results.

Claim 21

The method of claim 1 , wherein (a) additional terms are assigned to each document D in the collection, beyond the terms that occur in the document D, based upon the characteristics of the document D, without regard to the occurrence of the additional terms in the document D, and (b) the additional terms are placed in a segment S i of the document D with no other terms.

Claim 22

The method of claim 15 , wherein (a) additional terms are assigned to each document D in the collection, beyond the terms that occur in the document D, based upon the characteristics of the document D, without regard to the occurrence of the additional terms in the document D, and (b) the additional terms are placed in a segment S i of the document D with no other terms.

Claim 23

The method of claim 21 , wherein the additional terms are assigned automatically by (a) creating a search query Q comprised of terms in document D; (b) applying the search query Q to a collection of documents C 0 ; (c) selecting the N 0 documents from the collection of documents C 0 which achieve the highest scores upon application of the search query Q; and (d) selecting I T terms for automatic assignment from among terms in the N 0 documents based upon the co-occurrence of terms in the N 0 documents with terms in the document D.

Claim 24

The method of claim 22 , further comprising selecting the I T terms for automatic assignment by (a) calculating, for terms T k which occur in the N 0 documents selected, the co-occurrence C n (T j ,T k ) of the term T k with terms T j in document D: C ni ( T j ,T k )=log 10 ( co ki ( T j ,T k )+1)* idf ( T k )/log 10 ( N 0 ), where : co ki   ( T j , T k ) = ∑ n = 1 N 0   tf   ( T k , n ) * tf   ( T j , n ) tf (T k , n)=number of occurrences of term T k in document n in the N 0 documents, tf (T j , n)=number of occurrences of term T j in document n in the N 0 documents, idf (T k )=the inverted document frequency for the term T k , =min (1.0, log 10 ( N/N T )/5.0) N=the number of documents in the collection C 0 over which the query Q was run, N T =the number of documents in the collection C 0 over which the query Q was run, which contained the term T k , (b) calculating, for terms T k which occur in the N 0 documents selected, the co-occurrence f D (T k ) of that term T k with document D: f D   ( T k ) = ∏ j = 1 T 0   ( δ + C ni   ( T k , T j ) ) idf   ( T j )  where T j =a term in document D, T 0 =the number of terms in document D, idf (T j )=the inverted document frequency for the term T j , =min (1.0, log 10 ( N/N I )/5.0), N=the number of documents in the collection C 0 over which the query Q was run, N j =the number of documents in the collection C 0 over which the query Q was run, which contained the term T j and δ=a constant; and (c) selecting I T terms for automatic assignment from among the terms in the N 0 documents based upon the scores f D (T k ) achieved by the terms.

Claim 25

The method of claim 23 , wherein W STD , the weight assigned to term T in segment S I of document D, is f D (T k ) for all terms T automatically assigned.

Claim 26

The method of claim 24 , wherein W STD , the weight assigned to term T in segment S i of document D, is 1.0 for all terms T which occur in the document D.

Claim 27

The method of claim 23 , wherein the search query Q which is applied comprises all of the terms in document D.

Claim 28

The method of claim 23 , wherein the search query Q which is applied comprises all of the terms in document D with preselected stop terms eliminated.

Claim 29

The method of claim 23 , wherein the search query Q is applied to select documents from among the documents in the collection C 0 by calculating for each document D in the collection C 0 a score S D based upon the occurrence in the document D of terms in the search query Q.

Claim 30

The method of claim 28 , wherein in applying the search query Q to the collection of documents C 0 the total score S D for a document D in the collection C 0 is S D = ∑ T = 1 T 0   TF TD * IDF T where: T 0 is the number of terms in the search query Q, and TF TD is Robertson's term frequency for the Term T in the Document D = N TD /( N TD +K 1 +K 2 *( L D /L O )), where: N TD is the number of times the term T occurs in document D, L D is the length of document D, L O is the average length of documents in the collection C 0 to be searched, and K 1 and K 2 are constants and IDF T =log(( N+K 3 )/ N T )/log( N+K 4 ) where: N is the number of documents in the collection C 0 to be searched N T is the number of documents containing the term T in the collection C 0 to be searched, K 3 and K 4 are constants.

Claim 31

The method of claim 29 , wherein K 1 equals 0.5, K 2 equals 1.5, K 3 equals 0.5, and K 4 equals 1.0.

Claim 32

The method of claim 23 , wherein the number N 0 of documents chosen by application of the search query Q is predetermined.

Claim 33

The method of claim 31 , wherein the number N 0 is 50.

Claim 34

The method of claim 23 , wherein all documents whose scores upon application of the search query Q exceed a given cutoff score are selected.

Claim 35

The method of claim 23 , wherein co-occurrences are calculated for all terms contained in the N 0 documents selected.

Claim 36

The method of claim 23 , wherein co-occurrences are calculated for all terms contained in the N 0 documents selected, except that preselected stop terms are eliminated.

Claim 37

The method of claim 23 , wherein δ=0.01.

Claim 38

The method of claim 23 , wherein the number I T of terms automatically assigned is predetermined.

Claim 39

The method of claim 37 , wherein the number I T is 30.

Claim 40

The method of claim 23 , wherein all terms whose scores f D (T k ) exceed a given cutoff score are automatically assigned.

Claim 41

The method of claim 30 , wherein the number N 0 of documents chosen by application of the search query Q is 50, δ=0.01, and the number I T is 30.

Claim 44

The device of claim 41 , wherein the documents are Web pages.

Claim 45

The device of claim 41 , wherein the documents are Web sites.

Claim 46

The device of claim 41 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms used in a search query.

Claim 47

The device of claim 41 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms used in a search query, with preselected stop terms eliminated.

Claim 48

The device of claim 41 , wherein additional terms are chosen to be used in selecting documents which may be of interest based upon the terms used in the search query.

Claim 49

The device of claim 41 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms in a document identified as having characteristics that are desired in the documents to be selected.

Claim 50

The device of claim 41 , wherein the terms chosen to be used in selecting documents which may be of interest comprise the terms in a document identified as having characteristics that are desired in the documents to be selected, with preselected stop terms eliminated.

Claim 51

The device of claim 41 , wherein additional terms are chosen to be used in selecting documents based upon the terms which occur in the document identified as having characteristics that are desired in the documents to be selected.

Claim 52

The device of claim 41 , wherein the terms chosen to be used in selecting documents comprise the terms in a document being accessed by a user.

Claim 53

The device of claim 41 , wherein the terms chosen to be used in selecting documents comprise the terms in a document being accessed by a user, with preselected stop terms eliminated.

Claim 54

The device of claim 41 , wherein additional terms are chosen to be used in selecting documents based upon the terms which occur in the document being accessed by the user.

Claim 55

The device of claim 41 , wherein the documents in the collection are ranked in order of their generalized term frequency scores S D , and a predetermined number N of documents with the highest scores is selected.

Claim 56

The device of claim 41 , wherein all documents with generalized term frequency scores S D above a predetermined cutoff score S are selected.

Claim 57

The device of claim 41 , further comprising means for assigning a weight W SD to each segment S i of a plurality of documents D in the collection, wherein the generalized term frequency score S D for a document D is based upon Robertson's term frequency score, such that: S D = ∑ T = 1 T 0   ∑ S i = 1 S 0   TF STD * IDF ST where: S D is the total score for the document D, T 0 is the number of terms which occur in the collection of terms included in the search, S 0 is the number of segments in the document D, TF STD =Robertson's generalized term frequency for Term T in Segment S i of Document D = G STD /( G STD +K 1 +K 2 *W SD *( H SD /H SO )), where: G STD =the generalized term count for Term T in segment S i of Document D, = W SD *W STD *N STD , where: W SD is the weight assigned to segment S i of document D, W STD is the weight assigned to term T in segment S i of document D, and N STD is the number of times the term T occurs in segment S i of document D, H SD =the generalized length of segment S i of document D, H SD = ∑ T = 1 L SD   W STD * N STD where: L SD is the number of different terms in segment S i of document D, H SO =the generalized average length of segment S i of documents in the collection being searched, H SO = ∑ N = 1 N 0   ( W SD * H SD ) / ∑ N = 1 N 0   W SD , where: N 0 is the number of documents in the collection, and K 1 and K 2 are constants, IDF ST =the generalized inverted document frequency for term T, IDF ST =log(( N 0 +K 3 )/ N ST )/log( N 0 +K 4 ) where: N 0 is the number of documents in the collection N ST is the number of documents in the collection containing the term T in the segment S i , K 3 and K 4 are constants.

Claim 58

The device of claim 55 , wherein K 1 =0.5, K 2 =1.5, K 3 =0.5, and K 4 =1.0.

Claim 59

The device of claim 55 , wherein the weights W iD assigned to the i th segment of the documents in the collection are equal.

Claim 60

The device of claim 57 , wherein the weights W SD assigned to the segments of a document D in the collection have the property that ∑ S i = 1 S 0   W SD = 1.

Claim 61

The device of claim 57 , wherein the weights W iD are selected specifically for the collection of documents from which documents are to be chosen by carrying out test searches with different weights, and selecting for use the weights which yield the most useful results.

Claim 62

The device of claim 41 , further comprising (a) means for assigning additional terms to each document D in the collection, beyond the terms that occur in the document D, based upon the characteristics of the document D, without regard to the occurrence of the additional terms in the document D, and (b) means for placing the additional terms in a segment S i of the document D with no other terms.

Claim 63

The device of claim 55 , further comprising (a) means for assigning additional terms to each document D in the collection, beyond the terms that occur in the document D, based upon the characteristics of the document D, without regard to the occurrence of the additional terms in the document D, and (b) means for placing the additional terms in a segment S i of the document D with no other terms.

Claim 64

The device of claim 61 , wherein the means for assigning additional terms comprise: (a) means for creating a search query Q comprised of terms in document D; (b) means for applying the search query Q to a collection of documents C 0 ; (c) means for selecting the N 0 documents from the collection of documents C 0 which achieve the highest scores upon application of the search query Q; and (d) means for selecting I T terms for automatic assignment from among terms in the N 0 documents based upon the co-occurrence of the terms in the N 0 documents with the terms in the document D.

Claim 65

The device of claim 62 , wherein the means for selecting the I T terms for automatic assignment further comprise (a) means for calculating, for terms T k which occur in the N 0 documents selected, the co-occurrence C n (T j ,T k ) of the term T k with terms T j in document D: C ni ( T j ,T k )=log 10 ( co ki ( T j ,T k )+1)* idf ( T k )/log 10 ( N 0 ), where : co ki   ( T j , T k ) = ∑ n = 1 N 0   tf   ( T k , n ) * tf   ( T j , n ) tf (T k , n)=number of occurrences of term T k in document n in the N 0 documents, tf (T j , n)=number of occurrences of term T j in document n in the N 0 documents, idf (T k )=the inverted document frequency for the term T k , =min (1.0, log 10 ( N/N T )/5.0) N=the number of documents in the collection C 0 over which the query Q was run, N T =the number of documents in the collection C 0 over which the query Q was run, which contained the term T n (b) means for calculating, for terms T k which occur in the N 0 documents selected, the co-occurrence f D (T k ) of that term T k with document D: f D   ( T k ) = ∏ j = 1 T 0   ( δ + C ni   ( T k , T j ) ) idf   ( T j )  where T j =a term in document D, T 0 =the number of terms in document D, idf (T j )=the inverted document frequency for the term T j , =min (1.0, log 10 ( N/N I )/5.0), N=the number of documents in the collection C 0 over which the query Q was run, N J =the number of documents in the collection C 0 over which the query Q was run, which contained the term T j and δ=a constant; and (c) means for selecting I T terms for automatic assignment from among the terms in the N 0 documents based upon the scores f D (T k ) achieved by the terms.

Claim 66

The device of claim 63 , wherein W STD , the weight assigned to term T in segment S I of document D, is f D (T n ) for all terms T chosen to be index terms.

Claim 67

The device of claim 64 , wherein W STD , the weight assigned to term T in segment S i of document D, is 1.0 for all terms T which occur in the document D.

Claim 68

The device of claim 63 , wherein the search query Q which is applied comprises all of the terms in document D.

Claim 69

The device of claim 63 , wherein the search query Q which is applied comprises all of the terms in document D with preselected stop terms eliminated.

Claim 70

The device of claim 63 , wherein the search query Q is applied to select documents from among the documents in the collection C 0 by calculating for each document D in the collection C 0 a score S D based upon the occurrence in the document D of terms in the search query Q.

Claim 71

The device of claim 68 , wherein in applying the search query Q to the collection of documents C 0 the total score S D for a.document D in the collection C 0 is S D = ∑ T = 1 T 0  TF TD * IDF T where: T 0 is the number of terms in the search query Q, and TF TD is Robertson's term frequency for the Term T in the Document D = N TD /( N TD +K 1 +K 2 *( L D /L O )), where: N TD is the number of times the term T occurs in document D, L D is the length of document D, L O is the average length of documents in the collection C 0 to be searched, and K 1 and K 2 are constants and IDF T =log(( N+K 3 )/ N T )/log( N+K 4 ) where: N is the number of documents in the collection C 0 to be searched N T is the number of documents containing the term T in the collection C 0 to be searched, K 3 and K 4 are constants.

Claim 72

The device of claim 69 , wherein K 1 equals 0.5, K 2 equals 1.5, K 3 equals 0.5, and K 4 equals 1.0.

Claim 73

The device of claim 63 , wherein the number N 0 of documents chosen by application of the search query Q is predetermined.

Claim 74

The device of claim 71 , wherein the number N 0 is 50.

Claim 75

The device of claim 63 , wherein all documents whose scores upon application of the search query Q exceed a given cutoff score are selected.

Claim 76

The device of claim 63 , wherein co-occurrences are calculated for all terms contained in the N 0 documents selected.

Claim 77

The device of claim 63 , wherein co-occurrences are calculated for all terms contained in the N 0 documents selected, except that preselected stop terms are eliminated.

Claim 78

The device of claim 63 , wherein δ=0.01.

Claim 79

The device of claim 63 , wherein the number I T of terms automatically assigned is predetermined.

Claim 80

The device of claim 77 , wherein the number I T is 30.

Claim 81

The device of claim 63 , wherein all terms whose scores f D (T k ) exceed a given cutoff score are automatically assigned.

Claim 82

The device of claim 70 , wherein the number N 0 of documents chosen by application of the search query Q is 50, δ=0.01, and the number I T is 30.