Document information retrieval using global word co-occurrence patterns

Claim 3

The method of claim 1, wherein a matrix of word vectors is formed using all of the generated word vectors of the corpus.

Claim 4

The method of claim 2, wherein forming the matrix comprises the steps of: forming a first matrix from a first subset of words within the corpus, each element of the first matrix recording the number of times that two words within the first subset co-occur in the predetermined range; clustering the first matrix into groups to form a set of low coverage word clusters; forming a second matrix from the low coverage word clusters and a second subset of words within the corpus, the second subset containing more words than the first subset, each element of the second matrix recording a number of times that each word of the second subset co-occurs with each cluster of the set of low coverage word clusters within the predetermined range; clustering the second matrix into groups to form a set of high coverage word clusters; forming a third matrix from the high coverage word clusters and all of the words of the corpus, each element of the third matrix recording a number of times that each word of the corpus co-occurs with each cluster of the set of high coverage word clusters within the predetermined range; and reducing dimensionality of the third matrix to represent each element of the third matrix as a compact vector.

Claim 5

The method of claim 3, further comprising, before forming the first matrix: computing word frequency and word pair frequency for the words of the corpus; and removing words and word pairs with low frequency.

Claim 6

The method of claim 3, wherein a number of words selected from the corpus to form the first matrix is 3000 words.

Claim 7

The method of claim 5, wherein the selected words are medium frequency words.

Claim 8

The method of claim 3, wherein the clustering of the first matrix and the second matrix use a Buckshot fast clustering algorithm.

Claim 9

The method of claim 3, wherein a clustering algorithm is based on a cosine similarity between columns of the first matrix.

Claim 10

The method of claim 3, wherein a number of low coverage word clusters is 200.

Claim 11

The method of claim 3, wherein the second subset contains 20,000 words.

Claim 12

The method of claim 10, wherein the words within the second subset are the most frequent words excluding stop words.

Claim 13

The method of claim 3, wherein the reducing step uses a singular value decomposition method.

Claim 14

The method of claim 1, wherein the similarity between two words uses a cosine similarity function of: ##EQU18## where V is a set of words; w i is the word; w j is the co-occurring word; φ is a word encoding vector of the word and the co-occurring word.

Claim 15

The method of claim 1, wherein the word vector is generated based on: ##EQU19## where φ 1 is an order-1 vector for word w i ; tokens t j ,k are unique words in document d j within the corpus D; tokens t j ,l are neighbors of one of the tokens t j ,k ; W is a number of intervening words between tokens t j ,l and t j ,k ; and φ 0 is an order-0 vector for tokens t j ,l.

Claim 16

The method of claim 14, wherein the word vector is generated based on: ##EQU20## where φ 2 is an order-2 vector for word w i ; tokens t j ,k are unique words in document d j in the corpus D; v n is a frequent content word; tokens t j ,l are neighbors of one of the tokens t j ,k ; W is a number of intervening words between tokens t j ,l and t j ,k ; and φ 1 is an order-1 vector for token t j ,l.

Claim 17

The method of claim 15, wherein the number of intervening words is 50.

Claim 18

The method of claim 14, wherein the number of intervening words is 50.

Claim 19

The method of claim 1, further comprising generating a context vector for each document of the corpus, the context vector for a document based on the word vectors from the thesaurus for each word located in the document.

Claim 20

The method of claim 18, wherein generating the context vector comprises: extracting each word from the document; retrieving the word vector from the thesaurus for each extracted word; adding the retrieved word vectors together to form the context vector for the document.

Claim 21

The method of claim 19, wherein the word vectors are weighted before being added together.

Claim 22

The method of claim 19 further comprising the step of normalizing the context vector.

Claim 23

The method of claim 18, wherein generating the context vector uses the equation: ##EQU21## where d j is the vector for document j; w ij is the weight for word i in document j; and v i is the thesaurus vector for word i.

Claim 24

The method of claim 22 wherein weighting the words in the document is by using an augmented term frequency-inverse document frequency method: ##EQU22## where tf ij is the frequency of word i in document j; N is the total number of documents; and n i is the document frequency of word i.

Claim 25

The method of claim 22, wherein the context vector is normalized using the equation: ##EQU23## where p is the number of dimensions in the corpus; and d j is the context vector.

Claim 28

The method of claim 25, wherein computing the correlation coefficient uses the following formula: ##EQU24## where ψ(d i ) is the query vector and ψ(d j ) is the document vector.

Claim 29

The method of claim 25, wherein ranking of documents is from highest correlation coefficients to lowest correlation coefficients.

Claim 30

The method of claim 25, wherein the step of outputting comprises outputting, instead of the ranking for at least one document, at least one document according to its corresponding ranking.

Claim 31

The method of claim 25, further comprising outputting at least one document according to its ranking.

Claim 32

The method of claim 25, wherein the step of outputting comprises outputting, instead of the ranking for at least one document, at least one document according to its corresponding ranking.

Claim 33

The method of claim 25, further comprising outputting at least one document according to its ranking.

Claim 36

The method of claim 32, wherein generating the factor vectors comprises: retrieving the thesaurus word vectors for each word in the query; forming factor clusters of the query; generating a factor vector for each factor cluster.

Claim 37

The method of claim 32, wherein generating the factor vector uses the equation: ##EQU25## where f m is the factor vector for factor cluster m; w im is the weight for word i in factor cluster m; and v i is the thesaurus vector for word i.

Claim 38

The method of claim 32, wherein computing the correlation coefficient for each document uses the following formula: ##EQU26## where ψ(f m ) is the factor vector for factor cluster f m and ψ(d j ) is the context vector for document d j .

Claim 39

The method of claim 32, wherein ranking of the documents within a factor cluster is based on correlation: corr(f.sub.m,d.sub.j) where the rank of d j according to this ranking is r m (j); and corr(f m ,d j ) is the correlation of factor m and document j.

Claim 40

The method of claim 32, wherein computing the maximum rank for each document uses the following equation: r(j)=max.sub.m (r.sub.m (j)) where r(j) is the ranking of document j; and r m (j) is the rank of document j for factor cluster m.

Claim 41

The method of claim 32, wherein the rank of the most relevant document is output.

Claim 46

The apparatus of claim 40, wherein the thesaurus comprises: an extractor for retrieving a word from the corpus; a counter recording the number of times the word co-occurs with a co-occurring word located within a predetermined range, the co-occurring word being any word located before and after the word within the predetermined range, the counter recording the number for every co-occurring word within the predetermined range; and a generator generating a word vector for the word based on every recorded number.

Claim 47

The apparatus of claim 40, wherein the determining means ranks the documents based on the correlation relationship.

Claim 48

The apparatus of claim 40, wherein the processing means weights the word vectors before forming the document vectors and the query vector.

Claim 49

The apparatus of claim 40, wherein the memory is one of a RAM, a ROM and an external storage unit.

Claim 50

The apparatus of claim 40, wherein the query vector is a plurality of query factor vectors, each query factor vector based on a cluster factor of the query.

Claim 51

The apparatus of claim 45, wherein the determining means ranks the documents in factor clusters based on the correlation relationship between the document vectors and the query factor vectors of the query vector.

Claim 52

The apparatus of claim 46, wherein the relevant documents are determined by combining the ranking of the document within each factor cluster.