US6418433B1 — System and method for focussed web crawling

Claim 1

Independent Claim 128 words

A general purpose computer including a data storage device including a computer usable medium having computer readable code means for focussed crawling of a wide area computer network such as the World Wide Web, comprising: computer readable code means for receiving a seed set of Web pages in a crawl database, the seed set being selected based at least on relevance to at least one topic; computer readable code means for identifying outlink Web pages from one or more Web pages in the crawl database; computer readable code means for evaluating one or more outlink Web pages for relevance to the topic; and computer readable code means for causing outlinks only of Web pages evaluated as being relevant to the topic to be stored in the crawl database.

Claim 2

Dependent Claim 128 words

A general purpose computer including a data storage device including a computer usable medium having computer readable code means for focussed crawling of a wide area computer network such as the World Wide Web, comprising: computer readable code means for receiving a seed set of Web pages in a crawl database, the seed set being selected based at least on relevance to at least one topic; computer readable code means for identifying outlink Web pages from one or more Web pages in the crawl database; computer readable code means for evaluating one or more outlink Web pages for relevance to the topic; and computer readable code means for causing outlinks only of Web pages evaluated as being relevant to the topic to be stored in the crawl database.

Claim 3

Dependent Claim 27 words

The computer of claim 1 , further comprising: computer readable code means for assigning a revisitation priority to a Web page, based on the means for evaluating.

Claim 4

Dependent Claim 99 words

The computer of claim 1 , further comprising: at least one watchdog module periodically determining new and old pages to consider, the new pages being selected from the outlink Web pages, the old pages being selected from pages in the crawl database; and one or more worker modules responsive to the watchdog module for accessing the new and old pages to consider, each Web page being associated with a respective Num_Tries field representing the number of times the respective page has been accessed, the Num_Tries field of a Web page being incremented each time the Web page is considered.

Claim 5

Dependent Claim 90 words

The computer of claim 1 , further comprising one or more worker modules for accessing new and old pages to consider, each Web page being associated with a respective field representing the number of times the respective page has been accessed, the field of a Web page being incremented each time the Web page is considered, wherein outlinks of a Web page considered by the worker module are entered into a link table in the crawl database only when the means for evaluating determines that the Web page is relevant.

Claim 6

Dependent Claim 27 words

The computer of claim 4 , wherein the worker module includes means for determining whether a gathering rate of pages is considered is below a panic threshold.

Claim 7

Dependent Claim 41 words

The computer of claim 5 , wherein the computer includes computer readable code means for considering all outlinks and inlinks of at least one Web page in the crawl database when the gathering rate is at or below the panic threshold.

Claim 8

Dependent Claim 31 words

The computer of claim 5 , wherein the topic defines a scope, and the scope is increased to an expanded scope when the gathering rate is at or below the threshold.

Claim 9

Dependent Claim 45 words

The computer of claim 7 , further comprising computer readable code means for estimating a size of a prospective Web page consideration step based on the expanded scope, and permitting the prospective consideration step only when the size is at or below a size threshold.

Claim 10

Dependent Claim 83 words

A computer system for focussed searching of the World Wide Web, comprising: a computer including at least one worker module for undertaking work, the work including adding outlinks of a Web page to a crawl database only when the Web page is relevant to a predefined topic, the system including the crawl database, the crawl database being accessible by the computer, the crawl database being focussed only on the topic such that the system includes no comprehensive database of the World Wide Web.

Claim 11

Dependent Claim 83 words

A computer system for focussed searching of the World Wide Web, comprising: a computer including at least one worker module for undertaking work, the work including adding outlinks of a Web page to a crawl database only when the Web page is relevant to a predefined topic, the system including the crawl database, the crawl database being accessible by the computer, the crawl database being focussed only on the topic such that the system includes no comprehensive database of the World Wide Web.

Claim 12

Dependent Claim 99 words

The computer system of claim 9 , comprising: means executable by the computer for receiving into the crawl database a seed set of Web pages, the seed set being representative of at least the topic; means in the worker module for identifying outlink Web pages from one or more Web pages in the crawl database; means in the worker module for evaluating one or more outlink Web pages for relevance to the topic; and means in the worker module for causing outlinks of Web pages to be stored in the crawl database in response to the means for evaluating.

Claim 13

Dependent Claim 25 words

The computer system of claim 10 , further comprising: means for assigning a revisitation priority to a Web page, based on the means for evaluating.

Claim 14

Dependent Claim 102 words

The computer of claim 10 , further comprising a watchdog module for scheduling worker thread work, wherein the watchdog module periodically determines new and old pages to consider, the new pages being selected from the outlink Web pages, the old pages being selected from pages in the crawl database, the worker module being responsive to the watchdog module for accessing the new and old pages to consider, each Web page being associated with a respective field representing the number of times the respective page has been accessed, the field of a Web page being incremented each time the Web page is considered.

Claim 15

Dependent Claim 40 words

The computer of claim 12 , wherein outlinks of a Web page considered by the worker module are entered into a link table in the crawl database only when the means for evaluating determines that the Web page is relevant.

Claim 16

Dependent Claim 27 words

The computer of claim 12 , wherein the worker module includes means for determining whether a gathering rate of pages is considered is below a panic threshold.

Claim 17

Dependent Claim 39 words

The computer of claim 14 , wherein the worker module includes means for considering all outlinks and inlinks of at least one Web page in the crawl database when the gathering rate is at or below the panic threshold.

Claim 18

Dependent Claim 31 words

The computer of claim 14 , wherein the topic defines a scope, and the scope is increased to an expanded scope when the gathering rate is at or below the threshold.

Claim 19

Dependent Claim 46 words

The computer of claim 16 , wherein the worker module further comprises means for estimating a size of a prospective Web page consideration step based on the expanded scope, and permitting the prospective consideration step only when the size is at or below a size threshold.

Claim 20

Dependent Claim 118 words

A computer-implemented method for building a focussed database of Web pages, comprising the acts of: receiving a search query from a user; in response to the search query, accessing a crawl database containing only information pertaining to Web pages related to a predefined range of topics; periodically determining new and old pages to consider, the new pages being selected from outlink Web pages, the old pages being selected from pages in the crawl database; and accessing the new and old pages to consider, each Web page being associated with a respective field representing the number of times the respective page has been accessed, the field of a Web page being incremented each time the Web page is considered.

Claim 21

Dependent Claim 118 words

A computer-implemented method for building a focussed database of Web pages, comprising the acts of: receiving a search query from a user; in response to the search query, accessing a crawl database containing only information pertaining to Web pages related to a predefined range of topics; periodically determining new and old pages to consider, the new pages being selected from outlink Web pages, the old pages being selected from pages in the crawl database; and accessing the new and old pages to consider, each Web page being associated with a respective field representing the number of times the respective page has been accessed, the field of a Web page being incremented each time the Web page is considered.

Claim 22

Dependent Claim 99 words

The method of claim 18 , further comprising: initially receiving a seed set of Web pages in the crawl database, the seed set defining the range of topics; evaluating one or more Web pages in the database for relevance to the range of topics; identifying outlink Web pages from one or more Web pages in the crawl database in response to the evaluating step; evaluating one or more outlink Web pages for relevance to the range of topics; and causing outlinks only of Web pages evaluated as being relevant to the topics to be stored in the crawl database.

Claim 23

Dependent Claim 29 words

The method of claim 18 , wherein outlinks of a Web page are entered into a link table in the crawl database only when the Web page is relevant.

Claim 24

Dependent Claim 23 words

The method of claim 18 , further including determining whether a gathering rate of pages is considered to be below a panic threshold.

Claim 25

Dependent Claim 34 words

The method of claim 21 , further comprising considering all outlinks and inlinks of at least one Web page in the crawl database when the gathering rate is at or below the panic threshold.

Claim 26

Dependent Claim 31 words

The method of claim 21 , wherein the topic defines a scope, and the scope is increased to an expanded scope when the gathering rate is at or below the threshold.

Claim 27

Dependent Claim 40 words

The method of claim 23 , further comprising estimating a size of a prospective Web page consideration step based on the expanded scope, and permitting the prospective consideration step only when the size is at or below a size threshold.

Claim 28

Dependent Claim 96 words

A computer program device comprising: a computer program storage device readable by a digital processing apparatus; and a program means on the program storage device and including instructions executable by the digital processing apparatus for performing method steps for focussed Web crawling, the method steps comprising the acts of: receiving a search query from a user; in response to the search query, accessing a crawl database containing only information pertaining to Web pages related to a predefined topic or topics; and determining whether a gathering rate of pages is considered to be below a panic threshold.

Claim 29

Dependent Claim 96 words

A computer program device comprising: a computer program storage device readable by a digital processing apparatus; and a program means on the program storage device and including instructions executable by the digital processing apparatus for performing method steps for focussed Web crawling, the method steps comprising the acts of: receiving a search query from a user; in response to the search query, accessing a crawl database containing only information pertaining to Web pages related to a predefined topic or topics; and determining whether a gathering rate of pages is considered to be below a panic threshold.

Claim 30

Dependent Claim 78 words

The computer program device of claim 25 , wherein the acts further comprise: initially receiving a seed set of Web pages in the crawl database, the seed set defining the topic; identifying outlink Web pages from one or more Web pages in the crawl database; evaluating one or more outlink Web pages for relevance to the topic; and causing outlinks only of Web pages evaluated as being relevant to the topic to be stored in the crawl database.

Claim 31

Dependent Claim 86 words

The computer program device of claim 26 , wherein the acts further comprise: periodically determining new and old pages to consider, the new pages being selected from the outlink Web pages, the old pages being selected from pages in the crawl database; and accessing the new and old pages to consider, each Web page being associated with a respective field representing the number of times the respective page has been accessed, the field of a Web page being incremented each time the Web page is considered.

Claim 32

Dependent Claim 31 words

The computer program device of claim 26 , wherein outlinks of a Web page are entered into a link table in the crawl database only when the Web page is relevant.

Claim 33

Dependent Claim 39 words

The computer program device of claim 25 , wherein the acts further comprise considering all outlinks and inlinks of at least one Web page in the crawl database when the gathering rate is at or below the panic threshold.

Claim 34

Dependent Claim 33 words

The computer program device of claim 25 , wherein the topic defines a scope, and the scope is increased to an expanded scope when the gathering rate is at or below the threshold.

Claim 35

Dependent Claim 45 words

The computer program device of claim 30 , wherein the acts further comprise estimating a size of a prospective Web page consideration step based on the expanded scope, and permitting the prospective consideration step only when the size is at or below a size threshold.

Claim 36

Dependent Claim 101 words

A computer-implemented method for establishing a database of Web pages based on the relevance of the pages to at least one predefined topic, comprising the acts of: providing at least one example Web page; determining the relevance of at least one Web page based at least partially on the example Web page; and at least in part using the relevance of the Web page, determining whether to insert outlinks of the page into the database, wherein the act of determining whether to insert outlinks of the page into the database is undertaken using either a “soft” policy or a “hard” policy.

Claim 37

Dependent Claim 101 words

A computer-implemented method for establishing a database of Web pages based on the relevance of the pages to at least one predefined topic, comprising the acts of: providing at least one example Web page; determining the relevance of at least one Web page based at least partially on the example Web page; and at least in part using the relevance of the Web page, determining whether to insert outlinks of the page into the database, wherein the act of determining whether to insert outlinks of the page into the database is undertaken using either a “soft” policy or a “hard” policy.

Annuity Stage	Window Opens	Payment Deadline	Surcharge Period Until	Estimated Fee	Status
3.5 Years (Year 4)	2005-07-08	2006-01-07	2006-07-09	$2000.00	Window Closed / Lapsed
7.5 Years (Year 8)	2009-07-08	2010-01-07	2010-07-09	$3760.00	Window Closed / Lapsed
11.5 Years (Year 12)	2013-07-08	2014-01-07	2014-07-09	$7700.00	Window Closed / Lapsed

System and method for focussed web crawling

Abstract

Inventors

Bibliographic Details

Claims Analysis

References Cited (Backward Citations)

USPTO Annuity & Maintenance Windows

Assigned Categories & Tags

Assign Category Tag