US6792458B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for monitoring and analyzing internet traffic
Assignee: Google LLC
•
Filing: 2000-10-04
•
Granted: 2004-09-14
•
Expires: 2020-10-04
Abstract
Abstract A system and method for monitoring and analyzing Internet traffic is provided that is efficient, completely automated, and fast enough to handle the busiest websites on the Internet, processing data many times faster than existing systems. The system and method of the present invention processes data by reading log files produced by web servers, or by interfacing with the web server in real time, processing the data as it occurs. The system and method of the present invention can be applied to one website or thousands of websites, whether they reside on one server or multiple servers. The multi-site and sub-reporting capabilities of the system and method of the present invention makes it applicable to servers containing thousands of websites and entire on-line communities. In one embodiment, the system and method of the present invention includes e-commerce analysis and reporting functionality, in which data from standard traffic logs is received and merged with data from e-commerce systems. The system and method of the present invention can produce reports showing detailed "return on investment" information, including identifying which banner ads, referrals, domains, etc. are producing specific dollars.
Inventors
- 👤 Paul Nicolas Muret
- 👤 Hui Sok Moon
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-02 15:23
CPC Classifications
H; H04; H04L; H04L41/00; H04L41/22; G; G06; G06Q; G06Q30/00; G06Q30/06; H04L43/00; H04L43/06; H04L43/062; H04L41/14; H04L41/142; H04L43/04; H04L43/045; H04L43/065; H04L43/067; H04L43/10; H04L43/106
Claims Analysis
Total Claims: 85
Claim 1
Independent Claim
122 words
A system for analyzing and monitoring internet traffic, comprising: a relational database; and a log engine that processes log files received from at least one internet server and stores data processed from the log files in the relational database; wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location on the log file.
Claim 2
Independent Claim
122 words
A system for analyzing and monitoring internet traffic, comprising: a relational database; and a log engine that processes log files received from at least one internet server and stores data processed from the log files in the relational database; wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location on the log file.
Claim 3
Dependent Claim
16 words
The system of claim 1 , wherein the relational database comprises a plurality of hash tables.
Claim 4
Dependent Claim
52 words
The system of claim 1 , wherein the plurality of tables comprise: a visitor table that stores traffic information generated by a visitor to an internet site hosted by the at least one internet server; and a plurality of data tables, wherein each data table stores records related to a respective parameter.
Claim 5
Dependent Claim
29 words
The system of claim 3 , wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.
Claim 6
Dependent Claim
37 words
The system of claim 3 , wherein the respective parameters comprise: domain names from which the visitor originated; and web browsers used by the visitor; and other internet sites that referred the visitor to the internet site.
Claim 7
Independent Claim
163 words
A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising: a visitor centric database; and a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data derived from the hits in the visitor centric database, wherein the visitor centric database associates the traffic data derived from the hits with a visitor that generated the hit; wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location.
Claim 8
Independent Claim
163 words
A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising: a visitor centric database; and a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data derived from the hits in the visitor centric database, wherein the visitor centric database associates the traffic data derived from the hits with a visitor that generated the hit; wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location.
Claim 9
Dependent Claim
17 words
The system of claim 6 , wherein the visitor centric database comprises a plurality of hash tables.
Claim 10
Dependent Claim
54 words
The system of claim 6 , wherein the plurality of hash tables comprise: a visitor table that stores traffic information derived from the hits, wherein the visitor table contains a unique visitor record for each visitor; and a plurality of data tables, wherein each data table stores data related to a respective non-unique parameter.
Claim 11
Dependent Claim
29 words
The system of claim 8 , wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.
Claim 12
Dependent Claim
40 words
The system of claim 8 , wherein the respective non-unique parameters comprise: domain names from which the visitors originated; web browsers used by the visitors; and other internet sites that referred the visitors to the at least one internet site.
Claim 13
Dependent Claim
28 words
The system of claim 8 , wherein the log engine comprises a visitor identifier that determines if a hit originates from a new visitor or an existing visitor.
Claim 14
Dependent Claim
26 words
The system of claim 11 , wherein the visitor identifier is adapted to create a new visitor record if a hit originates from a new visitor.
Claim 15
Dependent Claim
29 words
The system of claim 6 , wherein the log engine comprises a database buffer that temporarily stores the traffic data derived from the hits logged in the log files.
Claim 16
Dependent Claim
31 words
The system of claim 11 , wherein the log engine further comprises a database updater that transfers the traffic data temporarily stored in the database buffer to the visitor centric database.
Claim 17
Dependent Claim
30 words
The system of claim 12 , wherein the database updater sorts the traffic data temporarily stored in the database buffer before transferring the traffic data to the visitor centric database.
Claim 18
Dependent Claim
30 words
The system of claim 6 , wherein the log engine comprises a log parser that reads log lines in the log files, and separates each log file into individual fields.
Claim 19
Dependent Claim
24 words
The system of claim 6 , further comprising a report engine that generates reports using the traffic data stored in the visitor centric database.
Claim 20
Dependent Claim
30 words
The system of claim 17 , wherein the report engine is adapted to generate reports that correlate money spent by a visitor to any other parameter of the traffic data.
Claim 21
Dependent Claim
31 words
The system of claim 17 , wherein the report engine is adapted to generate a top products report that ranks products purchased by visitors based on revenues generated by the products.
Claim 22
Dependent Claim
30 words
The system of claim 17 , wherein the report engine is adapted to generate at least one of a totals report, product tree report, regions report, and top scores report.
Claim 23
Dependent Claim
34 words
The system of claim 17 , wherein the report engine is adapted to generate a report that displays a value of at least one traffic data parameter over at least one predetermined time period.
Claim 24
Dependent Claim
29 words
The system of claim 21 , wherein the report comprises a snapshot report in which the at least one predetermined time period comprises seven consecutive 24 hour time periods.
Claim 25
Dependent Claim
32 words
The system of claim 21 , wherein the report comprises an hourly graph report in which the at least one predetermined time period comprises a plurality of consecutive one hour time periods.
Claim 26
Dependent Claim
31 words
The system of claim 17 , wherein the report engine is adapted to generate a top pages report that ranks website pages based on number of visitors to the website pages.
Claim 27
Dependent Claim
26 words
The system of claim 24 , wherein each entry in the top pages report comprises a link for accessing additional information about a respective website page.
Claim 28
Dependent Claim
27 words
The system of claim 17 , wherein the report engine is adapted to generate a search engine report that displays a list of most used search engines.
Claim 29
Dependent Claim
27 words
The system of claim 17 , wherein the report engine is adapted to generate a top domains report that displays regional and network information about the visitors.
Claim 30
Dependent Claim
35 words
The system of claim 17 , wherein the report engine is adapted to generate a browser tree report that ranks internet browsers based on which internet browsers are used most by visitors to a website.
Claim 31
Dependent Claim
30 words
The system of claim 28 , wherein each internet browser entry in the browser tree report includes a link for accessing information about different versions of a respective internet browser.
Claim 32
Dependent Claim
32 words
The system of claim 17 , wherein the report engine is adapted to generate a top entrances report that ranks starting points of visitors to a website based most used starting points.
Claim 33
Dependent Claim
94 words
The system of claim 17 , wherein the report engine is adapted to generate at least one of a summary report, a daily graph report, a monthly graph report, a top servers report, a file types report, a status/errors report, a posted forms report, a top referrals report, a top keywords report, a referral tree report, a domain tree report, a top countries report, a platform tree report, a top combos report, a top exits report, a click through report, a depth of visit report, a length of visit report, and a usernames report.
Claim 34
Dependent Claim
155 words
The system of claim 17 , wherein the report engine comprises: a template module that stores report templates; a session parser that receives report requests from the at least one server, and determines a type of report requested, data needed to generate a requested report and a format for the requested report; an authenticator that receives an identity of a report requester from the session parser, and verifies that the report requester has permission to view a requested report; a data query module that receives authentication information from the authenticator, and that queries the database for data needed to generate the requested report if the report requester has permission to view the requested report; and a format output module that receives the data needed to generate the requested report from the database, retrieves templates for the requested report from the template module, creates the requested report, and delivers the requested report to the report requester.
Claim 35
Dependent Claim
16 words
The system of claim 32 , wherein the template module also stores at least one dictionary.
Claim 36
Dependent Claim
28 words
The system of claim 33 , wherein the format output module is adapted to create the requested report in a selectable language using the at least one dictionary.
Claim 37
Dependent Claim
27 words
The system of claim 6 , wherein the log engine is configured to process hits from multiple internet sites that are logged to a single log file.
Claim 38
Dependent Claim
21 words
The system of claim 6 , wherein the log engine comprises a website identifier that identifies a source of each hit.
Claim 39
Dependent Claim
26 words
The system of claim 6 , wherein the log engine comprises a domain name system (DNS) resolver that determines host and domain information for each visitor.
Claim 40
Dependent Claim
24 words
The system of claim 37 , wherein the DNS resolver utilizes reverse DNS resolution to determine the host and domain information for each visitor.
Claim 41
Dependent Claim
26 words
The system of claim 6 , wherein the log engine is adapted to process e-commerce log files that contain information on money spent by a visitor.
Claim 42
Dependent Claim
193 words
An article of manufacture, comprising: a computer usable medium having computer readable program code embodied therein for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, the computer readable program code in the article of manufacture comprising: computer readable program code for receiving log files from the at least one internet server; computer readable program code for processing hits logged in each log file by: initiating a process loop when new data is present in the log file, during which the new data is processed and an “end of file” location is determined for use as a starting point for subsequent new data processing, and initiating a wait loop when new data is not present in the log file, wherein the wait loop delays data processing for a predetermined time interval before checking for new data in the log file; computer readable program code for storing traffic data derived from the hits in a database; and computer readable program code for associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.
Claim 43
Dependent Claim
193 words
An article of manufacture, comprising: a computer usable medium having computer readable program code embodied therein for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, the computer readable program code in the article of manufacture comprising: computer readable program code for receiving log files from the at least one internet server; computer readable program code for processing hits logged in each log file by: initiating a process loop when new data is present in the log file, during which the new data is processed and an “end of file” location is determined for use as a starting point for subsequent new data processing, and initiating a wait loop when new data is not present in the log file, wherein the wait loop delays data processing for a predetermined time interval before checking for new data in the log file; computer readable program code for storing traffic data derived from the hits in a database; and computer readable program code for associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.
Claim 44
Dependent Claim
17 words
The article of manufacture of claim 40 , wherein the database comprises a plurality of hash tables.
Claim 45
Dependent Claim
56 words
The article of manufacture of claim 41 , wherein the plurality of hash tables comprise: a visitor table that stores traffic information derived from the hits, wherein the visitor table contains a unique visitor record for each visitor; and a plurality of data tables, wherein each data table stores data related to a respective non-unique parameter.
Claim 46
Dependent Claim
31 words
The article of manufacture of claim 42 , wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.
Claim 47
Dependent Claim
44 words
The article of manufacture of claim 42 , wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for determining if a hit originates from a visitor with a preexisting visitor record in the database.
Claim 48
Dependent Claim
51 words
The article of manufacture of claim 44 , wherein the computer readable program code for determining if a hit originates from a visitor with a preexisting visitor record in the database creates a new visitor record if a hit originates from a visitor without a preexisting visitor record in the database.
Claim 49
Dependent Claim
29 words
The article of manufacture of claim 40 , further comprising computer readable program code for temporarily storing the traffic data derived from the hits logged in the log files.
Claim 50
Dependent Claim
40 words
The article of manufacture of claim 46 , wherein the computer readable program code for storing traffic data derived from the hits in a database comprises computer readable program code for transferring the temporarily stored traffic data to the database.
Claim 51
Dependent Claim
38 words
The article of manufacture of claim 47 , wherein the computer readable program code for transferring the temporarily stored traffic data to the database sorts the temporarily stored traffic data before transferring the traffic data to the database.
Claim 52
Dependent Claim
45 words
The article of manufacture of claim 40 , wherein the computer readable program code for processing hits logged in each log file further comprises computer readable program code for reading log lines in the log files, and for separating each log line into individual fields.
Claim 53
Dependent Claim
26 words
The article of manufacture of claim 49 , further comprising computer readable program code for generating reports using the associated traffic data stored in the database.
Claim 54
Dependent Claim
36 words
The article of manufacture of claim 40 , wherein the computer readable program code for processing hits logged in each log file processes hits originating from multiple internet sites and logged to a single log file.
Claim 55
Dependent Claim
34 words
The article of manufacture of claim 40 , wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for identifying a source of each hit.
Claim 56
Dependent Claim
36 words
The article of manufacture of claim 40 , wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for determining host and domain information for each visitor.
Claim 57
Dependent Claim
40 words
The article of manufacture of claim 53 , wherein the DNS resolver means computer readable program code for determining host and domain information for each visitor utilizes reverse DNS resolution to determine the host and domain information for each visitor.
Claim 58
Independent Claim
331 words
A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising: a database; a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data extracted from the processed hits in the database, wherein the log engine comprises, a database buffer that temporarily stores traffic data received from the database, a log parser that processes each hit in each log file, and separates each hit into its individual fields, wherein the log parser, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time periods to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location. a visitor identifier that receives each hit's individual fields from the log parser, identifies each hit as originating from either a new visitor or an existing visitor, and creates a new visitor record in the database buffer if a hit originates from a new visitor, a buffer updater that, prior to processing a new log file, copies previously stored data from the database to the database buffer, and wherein, for each hit, the buffer updater locates in the database buffer the visitor record identified or created by the visitor identifier for a respective hit, and updates the identified or created visitor record in the database buffer with traffic data derived from the respective hit, and a database updater that copies updated traffic data from the database buffer to the database after all hits in the new log file have been processed; and a report engine that generates reports using the traffic data stored in the database.
Claim 59
Independent Claim
331 words
A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising: a database; a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data extracted from the processed hits in the database, wherein the log engine comprises, a database buffer that temporarily stores traffic data received from the database, a log parser that processes each hit in each log file, and separates each hit into its individual fields, wherein the log parser, when new log file data is present in the log file, processes said new log file data and determines an “end of file” location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time periods to check for new log file data, and commences processing of any new log file data at a most recent determined “end of file” location. a visitor identifier that receives each hit's individual fields from the log parser, identifies each hit as originating from either a new visitor or an existing visitor, and creates a new visitor record in the database buffer if a hit originates from a new visitor, a buffer updater that, prior to processing a new log file, copies previously stored data from the database to the database buffer, and wherein, for each hit, the buffer updater locates in the database buffer the visitor record identified or created by the visitor identifier for a respective hit, and updates the identified or created visitor record in the database buffer with traffic data derived from the respective hit, and a database updater that copies updated traffic data from the database buffer to the database after all hits in the new log file have been processed; and a report engine that generates reports using the traffic data stored in the database.
Claim 60
Dependent Claim
22 words
The system of claim 55 , wherein the log engine further comprises a website identifier that identifies a source of each hit.
Claim 61
Dependent Claim
28 words
The system of claim 56 , wherein the website identifier identifies the source of each hit from website identifier text received from the log parser for each hit.
Claim 62
Dependent Claim
31 words
The system of claim 55 , wherein the log engine further comprises a domain name system (DNS) resolver that determines host and domain information for each visitor to an internet site.
Claim 63
Dependent Claim
19 words
The system of claim 58 , wherein the DNS resolver is adapted to process multiple DNS queries in parallel.
Claim 64
Dependent Claim
26 words
The system of claim 55 , wherein the log engine is adapted to process e-commerce log files that contain information on money spent by the visitor.
Claim 65
Independent Claim
161 words
A method of analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising the steps of: receiving log files from the at least one internet server; processing hits logged in each log file, as each hit is logged to each log file, by: (a) processing new hits present in a log file, (b) determining an “end of file” location on the log file, (c) waiting for a predetermined time period if no new hits are present in the log file, (d) checking for new hits in the log file after the predetermined time period, and (e) processing any new hits discovered in the log file by starting at the determined end of file location in the log file; storing, in a database, traffic data derived from the hits; and associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.
Claim 66
Independent Claim
161 words
A method of analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising the steps of: receiving log files from the at least one internet server; processing hits logged in each log file, as each hit is logged to each log file, by: (a) processing new hits present in a log file, (b) determining an “end of file” location on the log file, (c) waiting for a predetermined time period if no new hits are present in the log file, (d) checking for new hits in the log file after the predetermined time period, and (e) processing any new hits discovered in the log file by starting at the determined end of file location in the log file; storing, in a database, traffic data derived from the hits; and associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.
Claim 67
Dependent Claim
22 words
The method of claim 61 , further comprising the step of generating reports using the associated traffic data stored in the database.
Claim 68
Dependent Claim
22 words
The method of claim 62 , wherein the traffic data derived from the hits is stored in a plurality of hash tables.
Claim 69
Dependent Claim
45 words
The method of claim 63 , wherein traffic information derived from the hits are stored in a visitor hash table that contains a unique visitor record for each visitor, and wherein data related to at least one non-unique parameter is stored in respective data tables.
Claim 70
Dependent Claim
32 words
The method of claim 64 , wherein the visitor hash table comprises at least one pointer that points to at least one record stored in at least one of the data tables.
Claim 71
Dependent Claim
48 words
The method of claim 64 , further comprising the steps of: determining if a hit originates from a visitor with a preexisting visitor record in the database; and creating a new visitor record if the hit originates from a visitor without a preexisting visitor record in the database.
Claim 72
Dependent Claim
32 words
The method of claim 61 , further comprising the step of temporarily storing the traffic data derived from the hits in a buffer prior to storing the traffic data in the database.
Claim 73
Dependent Claim
28 words
The method of claim 67 , further comprising the step of sorting the traffic data stored in the buffer prior to storing the traffic data in the database.
Claim 74
Dependent Claim
36 words
The method of claim 61 , wherein the step of processing hits logged in each log file comprises the steps of: reading log lines in the log files; and separating each log line into individual fields.
Claim 75
Dependent Claim
28 words
The method of claim 69 , wherein the hits logged in each log file are processed in real time as each hit is logged to a log file.
Claim 76
Dependent Claim
19 words
The method of claim 61 , further comprising the step of identifying a source from which each hit originates.
Claim 77
Dependent Claim
19 words
The method of claim 61 , further comprising the step of determining host and domain information for each visitor.
Claim 78
Dependent Claim
23 words
The method of claim 72 , wherein host and domain information for each visitor is determined using reverse domain name system (DNS) resolution.
Claim 79
Independent Claim
212 words
A method of processing a log file to obtain traffic data, comprising the steps of: copying previously stored traffic data from a database to a database buffer; separating hits logged in the log file into individual fields, wherein each hit is processed as it is logged to the log file by: (a) processing new hits present in the log file, (b) determining an “end of file” location in the log file, (c) waiting for a predetermined time period if no new hits are present in the log file, (d) checking for new hits in the log file after the predetermined time period, (e) processing any new hits discovered in the log file by starting at the end of file location in the log file, and (f) repeating steps (a)-(e); identifying each hit as originating from either a new visitor or an existing visitor; creating a new visitor record in the database buffer if a hit originates from a new visitor; for each hit, locating the visitor record identified or created and updating the identified or created visitor record in the database buffer with traffic data derived from the respective hit; and copying updated traffic data from the database buffer to the database after all hits in the log file have been processed.
Claim 80
Independent Claim
212 words
A method of processing a log file to obtain traffic data, comprising the steps of: copying previously stored traffic data from a database to a database buffer; separating hits logged in the log file into individual fields, wherein each hit is processed as it is logged to the log file by: (a) processing new hits present in the log file, (b) determining an “end of file” location in the log file, (c) waiting for a predetermined time period if no new hits are present in the log file, (d) checking for new hits in the log file after the predetermined time period, (e) processing any new hits discovered in the log file by starting at the end of file location in the log file, and (f) repeating steps (a)-(e); identifying each hit as originating from either a new visitor or an existing visitor; creating a new visitor record in the database buffer if a hit originates from a new visitor; for each hit, locating the visitor record identified or created and updating the identified or created visitor record in the database buffer with traffic data derived from the respective hit; and copying updated traffic data from the database buffer to the database after all hits in the log file have been processed.
Claim 81
Dependent Claim
22 words
The method of claim 74 , further comprising the step of generating a report based on the traffic data in the database.
Claim 82
Dependent Claim
18 words
The method of claim 74 , wherein hits originating from multiple sources are logged to the log file.
Claim 83
Dependent Claim
19 words
The method of claim 76 , further comprising the step of identifying a source from which each hit originates.
Claim 84
Dependent Claim
19 words
The method of claim 74 , further comprising the step of determining host and domain information for each visitor.
Claim 85
Dependent Claim
23 words
The method of claim 78 , wherein host and domain information for each visitor is determined using reverse domain name system (DNS) resolution.
References Cited (Backward Citations)
US5689416A
Cited
US5511190A
Cited
US5675510A
Cited
US5787253A
Cited
US5727129A
Cited
US5732218A
Cited
US6112238A
Cited
US6662227B2
Cited
US5796952A
Cited
US5878223A
Cited
US5951642A
Cited
US6122639A
Cited
US6317787B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-09-14 | 2008-03-15 | 2008-09-14 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-09-14 | 2012-03-15 | 2012-09-14 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-09-14 | 2016-03-15 | 2016-09-14 | $7700.00 | Window Closed / Lapsed |