US6286006B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for finding mirrored hosts by analyzing urls
Assignee: R2 Solutions LLC
•
Filing: 1999-05-07
•
Granted: 2001-09-04
•
Expires: 2019-05-07
Abstract
Abstract A method and apparatus that detects mirrored host pairs using information about a large set of pages, including URLs. The identities of the detected mirrored hosts are then saved so that browsers, crawlers, proxy servers, or the like can correctly identify mirrored web sites. The described embodiments of the present invention look at the URLs of pages hosts to determine whether the hosts are potentially mirrored.
Inventors
- 👤 Krishna A. Bharat
- 👤 Andrei Broder
- 👤 Steven C. Glassman
- 👤 Jeffrey Dean
- 👤 Monika R. Henzinger
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-08 18:23
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/90; G06F16/95; G06F16/957; G06F16/9574; Y; Y10; Y10S; Y10S707/00; Y10S707/99931; Y10S707/99933; Y10S707/99935
Claims Analysis
Total Claims: 24
Claim 1
Independent Claim
74 words
A method of determining mirrored web hosts, comprising: receiving information about the addresses of a plurality of web sites stored on a plurality of hosts; determining a plurality of terms of the URLs associated with every host; weighting the terms in inverse proportion to frequency; determining a similarity score for host pair in accordance with the weighted terms; and outputting a list of potential pairs of mirrored hosts in accordance with their similarity scores.
Claim 2
Independent Claim
74 words
A method of determining mirrored web hosts, comprising: receiving information about the addresses of a plurality of web sites stored on a plurality of hosts; determining a plurality of terms of the URLs associated with every host; weighting the terms in inverse proportion to frequency; determining a similarity score for host pair in accordance with the weighted terms; and outputting a list of potential pairs of mirrored hosts in accordance with their similarity scores.
Claim 3
Dependent Claim
20 words
The method of claim 1 , wherein the terms include “n-grams” having two or more contiguous tokens from the URLs.
Claim 4
Dependent Claim
27 words
The method of claim 1 , wherein the terms include “n-grams” having two or more continuous tokens from the URLs and URL depth information for the tokens.
Claim 5
Dependent Claim
14 words
The method of claim 1 , wherein the terms include paths of the URLs.
Claim 6
Dependent Claim
17 words
The method of claim 1 , wherein the terms include substrings of the paths of the URLs.
Claim 7
Dependent Claim
23 words
The method of claim 1 , wherein the terms are placed in a <term,host> tuple and the tuples are sorted by the terms.
Claim 8
Dependent Claim
14 words
The method of claim 1 , wherein the terms include prefixes of the URLs.
Claim 9
Dependent Claim
12 words
The method of claim 1 , wherein the similarity scores are normalized.
Claim 10
Dependent Claim
28 words
The method of claim 1 , further comprising accessing pages on each host corresponding to paths selected from both hosts to determine whether the hosts are mirrored hosts.
Claim 11
Dependent Claim
25 words
The method of claim 9 , wherein the hosts are mirrored if at least a predetermined number of path matches occurs for the selected paths.
Claim 12
Dependent Claim
68 words
The method of claim 1 , further comprising: attempting to access selected pages on each host of a host pair, the pages corresponding to paths selected from each host, to determine whether the hosts are mirrored hosts; categorizing the pages as having various matching categories; and categorizing the host pairs in one of a plurality of similarity categories in accordance with the matching categories of the selected pages.
Claim 13
Dependent Claim
57 words
The method of claim 11 , wherein the matching categories indicate one of: access of the path on a source host failed; access of the path on a target host failed; content is byte-wise identical; documents are 100% similar after removal of; common content above a threshold for high similarity; and path is valid but no similarity.
Claim 14
Dependent Claim
25 words
The method of claim 11 , wherein the host pairs are divided into five similarity categories in accordance with the matching categories of the pages.
Claim 15
Independent Claim
89 words
An apparatus that determines mirrored web hosts, comprising: software configured to receive information about the addresses of a plurality of web sites stored on a plurality of hosts; software configured to determine a plurality of terms of the URLs associated with every host; software configured to weight the terms in inverse proportion to frequency; software configured to determine a similarity score for host pair in accordance with the weighted terms; and software configured to output a list of potential pairs of mirrored hosts in accordance with their similarity scores.
Claim 16
Independent Claim
89 words
An apparatus that determines mirrored web hosts, comprising: software configured to receive information about the addresses of a plurality of web sites stored on a plurality of hosts; software configured to determine a plurality of terms of the URLs associated with every host; software configured to weight the terms in inverse proportion to frequency; software configured to determine a similarity score for host pair in accordance with the weighted terms; and software configured to output a list of potential pairs of mirrored hosts in accordance with their similarity scores.
Claim 17
Dependent Claim
19 words
The apparatus of claim 14 , wherein the terms include “n-grams” having pairs of contiguous tokens from the URLs.
Claim 18
Dependent Claim
26 words
The apparatus of claim 14 , wherein the terms include “n-grams” having pairs of continuous tokens from the URLs and URL depth information for the tokens.
Claim 19
Dependent Claim
14 words
The apparatus of claim 14 , wherein the terms include paths of the URLs.
Claim 20
Dependent Claim
17 words
The apparatus of claim 14 , wherein the terms include substrings of the paths of the URLS.
Claim 21
Dependent Claim
14 words
The apparatus of claim 14 , wherein the terms include prefixes of the URLs.
Claim 22
Dependent Claim
12 words
The apparatus of claim 14 , wherein the similarity scores are normalized.
Claim 23
Dependent Claim
101 words
A computer program product, comprising: computer program code devices configured to receive information about the addresses of a plurality of web sites stored on a plurality of hosts; computer program code devices configured to determine a plurality of terms of the URLs associated with every host; computer program code devices configured to weight the terms in inverse proportion to frequency; computer program code devices configured to determine a similarity score for host pair in accordance with the weighted terms; and computer program code devices configured to output a list of potential pairs of mirrored hosts in accordance with their similarity scores.
Claim 24
Dependent Claim
101 words
A computer program product, comprising: computer program code devices configured to receive information about the addresses of a plurality of web sites stored on a plurality of hosts; computer program code devices configured to determine a plurality of terms of the URLs associated with every host; computer program code devices configured to weight the terms in inverse proportion to frequency; computer program code devices configured to determine a similarity score for host pair in accordance with the weighted terms; and computer program code devices configured to output a list of potential pairs of mirrored hosts in accordance with their similarity scores.
References Cited (Backward Citations)
US6154744A
Cited
US5935207A
Cited
US5909677A
Cited
US6105019A
Cited
US6167398A
Cited
US5895470A
Cited
US5978797A
Cited
US6119124A
Cited
US5991714A
Cited
US6138113A
Cited
WO2000068838A2
Cited
US6125395A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2004-09-03 | 2005-03-05 | 2005-09-04 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2008-09-03 | 2009-03-05 | 2009-09-04 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2012-09-03 | 2013-03-05 | 2013-09-04 | $7700.00 | Window Closed / Lapsed |