US6145003A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method of web crawling utilizing address mapping
Assignee: Microsoft Technology Licensing LLC
•
Filing: 1997-12-17
•
Granted: 2000-11-07
•
Expires: 2017-12-17
Abstract
Abstract A computer-based system and method of retrieving information pertaining to Web documents on a computer network is disclosed. The method includes maintaining an address map that associates primary addresses with secondary addresses. A primary address includes a network retrieval protocol and a network address. The secondary address may include a different retrieval protocol or a different network address from the primary document address. A Web crawler retrieves a Web document using the primary document address, and determines whether the address map contains a secondary document address prefix corresponding to the primary document address prefix. If a secondary document address prefix exists, the Web crawler creates a secondary address, retrieves additional information pertaining to the Web document, and combines the additional information with the data retrieved from the Web document. The combined data may be stored in an index, and subsequently used to perform a document search.
Inventors
- 👤 Sankrant Sanu
- 👤 Dmitriy Meyerzon
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-29 18:40
CPC Classifications
H; H04; H04L; H04L67/00; H04L67/01; H04L67/10; H04L61/00; H04L61/45; H04L69/00; H04L69/30; H04L69/32; H04L69/322; H04L69/329; H04L67/02; Y; Y10; Y10S; Y10S707/00; Y10S707/99931; Y10S707/99933
Claims Analysis
Total Claims: 31
Claim 1
Independent Claim
89 words
A computer-based method of retrieving Web document information from a computer network, comprising: retrieving a Web document from a computer network using a first protocol included in a primary document address specification; obtaining data from the Web document; determining whether the primary document address specification has a corresponding secondary document address specification; and if the primary document address specification has a corresponding secondary document address specification, retrieving supplementary data from the computer network pertaining to the Web document using a second protocol included in the secondary document address specification.
Claim 2
Dependent Claim
89 words
A computer-based method of retrieving Web document information from a computer network, comprising: retrieving a Web document from a computer network using a first protocol included in a primary document address specification; obtaining data from the Web document; determining whether the primary document address specification has a corresponding secondary document address specification; and if the primary document address specification has a corresponding secondary document address specification, retrieving supplementary data from the computer network pertaining to the Web document using a second protocol included in the secondary document address specification.
Claim 3
Dependent Claim
46 words
The method of claim 1, wherein the primary document address specification includes the first protocol and a first network address, and the secondary document address specification includes the second protocol and a second network address, and wherein the first protocol is different from the second protocol.
Claim 4
Dependent Claim
17 words
The method of claim 2, wherein the first protocol is HTTP and the second protocol is FILE.
Claim 5
Dependent Claim
24 words
The method of claim 3, further comprising retrieving an access control list corresponding to the Web document by using the secondary document address specification.
Claim 6
Dependent Claim
49 words
The method of claim 4, further comprising storing the supplementary data pertaining to the Web document retrieved using the second protocol from the secondary document address specification with the data retrieved from the Web document using the first protocol from the primary document address specification in a document index.
Claim 7
Dependent Claim
47 words
The method of claim 1, wherein the primary document address specification includes the first protocol and a first network address, and the secondary document address specification includes the second protocol and a second network address, and the first network address is different from the second network address.
Claim 8
Dependent Claim
36 words
The method of claim 6, wherein retrieving supplementary data pertaining to the Web document by using the secondary document address specification comprises retrieving a second Web document that includes supplementary data pertaining to the Web document.
Claim 9
Dependent Claim
24 words
The method of claim 7, wherein retrieving the second Web document includes using the hypertext transfer protocol (HTTP) to retrieve the second Web document.
Claim 10
Dependent Claim
22 words
The method of claim 7, wherein retrieving the second Web document includes using a database specification to retrieve the second Web document.
Claim 11
Dependent Claim
37 words
The method of claim 1, wherein determining whether the primary document address specification has a corresponding secondary document address specification includes determining whether an entry corresponding to the primary document address specification exists in an address map.
Claim 12
Dependent Claim
26 words
The method of claim 10, wherein the entry corresponding to the primary document address specification includes a transfer protocol specification and a top level domain specification.
Claim 13
Dependent Claim
86 words
The method of claim 1, further comprising: determining whether the primary document address specification has a corresponding tertiary document address specification; if the primary document address specification has a corresponding tertiary document address specification, retrieving further data pertaining to the Web document by using the tertiary document specification; and if the primary document address specification has a corresponding tertiary document address specification, storing the further data pertaining to the Web document obtained using the tertiary document address specification with the data obtained from the Web document.
Claim 14
Dependent Claim
31 words
The method of claim 1, wherein the secondary document address specification is automatically built by replacing a secondary address prefix for a primary address prefix in the primary document address specification.
Claim 15
Dependent Claim
140 words
The method of claim 13, further comprising: (a) obtaining a URL from a transaction log; (b) parsing the URL into a URL prefix and URL suffix; (c) providing an address map containing a plurality of primary address prefixes and corresponding secondary address prefixes; (d) determining if the URL prefix is included in the address map as a primary address prefix; (i) if the URL prefix is included in the address map as a primary address prefix, combining a secondary address prefix that corresponds to the primary address prefix with the URL suffix to build the secondary document address specification; and (ii) if the URL prefix is not included in the address map as a primary address prefix, changing the parsing of the URL to incrementally reduce the URL prefix and increase the URL suffix and then repeating this paragraph (d).
Claim 16
Dependent Claim
170 words
A computer-based method of retrieving information from a computer network during a network crawl, comprising: retrieving an electronic document from the computer network, the electronic document including at least one hyperlink specification including a primary document address specification; retrieving at least one primary document address specification from the electronic document using a first protocol included in the primary address specification, each primary document address corresponding to a linked electronic document; determining whether the primary document address specification has a corresponding secondary document address specification; if the primary document address specification has a corresponding secondary document address specification, retrieving supplementary data pertaining to the linked electronic document from the computer network using a second protocol included in the secondary document address specification; and if the primary document address specification has a corresponding secondary document address specification, storing the supplementary data pertaining to the linked electronic document obtained using the secondary document address specification and associating the stored supplementary data pertaining to the linked electronic document with the primary document address specification.
Claim 17
Dependent Claim
170 words
A computer-based method of retrieving information from a computer network during a network crawl, comprising: retrieving an electronic document from the computer network, the electronic document including at least one hyperlink specification including a primary document address specification; retrieving at least one primary document address specification from the electronic document using a first protocol included in the primary address specification, each primary document address corresponding to a linked electronic document; determining whether the primary document address specification has a corresponding secondary document address specification; if the primary document address specification has a corresponding secondary document address specification, retrieving supplementary data pertaining to the linked electronic document from the computer network using a second protocol included in the secondary document address specification; and if the primary document address specification has a corresponding secondary document address specification, storing the supplementary data pertaining to the linked electronic document obtained using the secondary document address specification and associating the stored supplementary data pertaining to the linked electronic document with the primary document address specification.
Claim 18
Dependent Claim
46 words
The method of claim 15, wherein the primary document address specification includes the first protocol and a first network address, and the secondary document address specification includes the second protocol and a second network address, and wherein the first protocol is different from the second protocol.
Claim 19
Dependent Claim
47 words
The method of claim 15, wherein the primary document address specification includes the first protocol and a first network address, and the secondary document address specification includes the second protocol and a second network address, and the first network address is different from the second network address.
Claim 20
Dependent Claim
37 words
The method of claim 15, wherein determining whether the primary document address specification has a corresponding secondary document address specification includes determining whether an entry corresponding to the primary document address specification exists in an address map.
Claim 21
Dependent Claim
68 words
The method of claim 15, further comprising: retrieving data from the linked electronic document using the primary document address specification; storing the data retrieved from the linked electronic document using the primary document address specification; and associating the data retrieved from the linked electronic document using the primary document address specification with the supplementary data pertaining to the linked electronic document retrieved using the secondary document address specification.
Claim 22
Dependent Claim
63 words
The method of claim 15, further comprising: automatically retrieving a plurality of primary document address specifications from a plurality of hyperlinks included in the electronic document; automatically retrieving a plurality of secondary document address specifications corresponding to said plurality of primary document address specifications; and automatically retrieving supplementary data pertaining to the linked electronic document using said plurality of secondary document address specifications.
Claim 23
Dependent Claim
30 words
The method of claim 15, wherein the secondary address specification is automatically built by replacing a secondary address prefix for a primary address prefix in the primary document address specification.
Claim 24
Dependent Claim
139 words
The method of claim 21, further comprising: (a) obtaining a URL from a transaction log; (b) parsing the URL into a URL prefix and URL suffix; (c) providing an address map containing a plurality of primary address prefixes and corresponding secondary address prefixes; (d) determining if the URL prefix is included in the address map as a primary address prefix; (i) if the URL prefix is included in the address map as a primary address prefix, combining a secondary address prefix that corresponds to the primary address prefix with the URL suffix to build the secondary address specification; and (ii) if the URL prefix is not included in the address map as a primary address prefix, changing the parsing of the URL to incrementally reduce the URL prefix and increase the URL suffix and then repeating this paragraph (d).
Claim 25
Independent Claim
236 words
A system for performing a Web crawl, the system comprising: a server computer having a Web crawler program executing thereon; an address map accessible to the Web crawler program and containing a plurality of primary Web addresses and a plurality of secondary Web addresses, each primary Web address having a corresponding secondary Web address; the primary Web address including a first protocol for the retrieval of a Web document at the primary Web address; the secondary Web address including a second protocol for the retrieval of a Web document at the secondary Web address; the second protocol for the retrieval of a Web document at the secondary Web address being different than the first protocol for the retrieval of a Web document at the primary Web address; a computer network including at least one Web server having a plurality of Web documents stored thereon, each Web document having a corresponding primary Web address; a database containing information pertaining to the plurality of Web documents; program code for: retrieving a primary Web address corresponding to one of the Web documents; determining whether the primary Web address has a corresponding secondary Web address; selectively retrieving supplementary information pertaining to said one of the Web documents using the corresponding secondary Web address; and if the supplementary information pertaining to said one of the Web documents is retrieved using the secondary Web address, storing said supplementary information in the database.
Claim 26
Independent Claim
236 words
A system for performing a Web crawl, the system comprising: a server computer having a Web crawler program executing thereon; an address map accessible to the Web crawler program and containing a plurality of primary Web addresses and a plurality of secondary Web addresses, each primary Web address having a corresponding secondary Web address; the primary Web address including a first protocol for the retrieval of a Web document at the primary Web address; the secondary Web address including a second protocol for the retrieval of a Web document at the secondary Web address; the second protocol for the retrieval of a Web document at the secondary Web address being different than the first protocol for the retrieval of a Web document at the primary Web address; a computer network including at least one Web server having a plurality of Web documents stored thereon, each Web document having a corresponding primary Web address; a database containing information pertaining to the plurality of Web documents; program code for: retrieving a primary Web address corresponding to one of the Web documents; determining whether the primary Web address has a corresponding secondary Web address; selectively retrieving supplementary information pertaining to said one of the Web documents using the corresponding secondary Web address; and if the supplementary information pertaining to said one of the Web documents is retrieved using the secondary Web address, storing said supplementary information in the database.
Claim 27
Dependent Claim
18 words
The system of claim 23, further comprising a search engine that performs a Web search using the database.
Claim 28
Dependent Claim
53 words
The system of claim 24, wherein the first protocol is HTTP, the second protocol is FILE, and the retrieving of supplementary information pertaining to said one of the Web documents using the secondary Web address includes using file system commands to retrieve the supplementary information pertaining to said one of the Web documents.
Claim 29
Dependent Claim
22 words
The system of claim 25, wherein the supplementary information pertaining to said one of the Web documents includes an access control list.
Claim 30
Dependent Claim
39 words
The system of claim 23, wherein each primary Web address includes a data transfer protocol specification and a top level domain specification, and each secondary Web address includes a data transfer protocol specification and a top level domain specification.
Claim 31
Dependent Claim
19 words
The system of claim 23, wherein the address map and said plurality of Web documents reside on different computers.
References Cited (Backward Citations)
US5649186A
Cited
US5793966A
Cited
US5974455A
Cited
US5870546A
Cited
US5855020A
Cited
US5862325A
Cited
US5864852A
Cited
US5903892A
Cited
US5892908A
Cited
US5875446A
Cited
US5895470A
Cited
US5987454A
Cited
US5933822A
Cited
US5848410A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-11-07 | 2004-05-08 | 2004-11-07 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-11-07 | 2008-05-08 | 2008-11-07 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-11-07 | 2012-05-08 | 2012-11-07 | $7700.00 | Window Closed / Lapsed |