US6718486B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Fault monitor for restarting failed instances of the fault monitor
Assignee: Individual
•
Filing: 2000-07-24
•
Granted: 2004-04-06
•
Expires: 2020-07-24
Abstract
Abstract A computer system having a fault-tolerance framework in an extendable computer architecture. The computer system is formed of clusters of nodes where each node includes computer hardware and operating system software for executing jobs that implement the services provided by the computer system. Jobs are distributed across the nodes under control of a hierarchical resource management unit. The resource management unit includes hierarchical monitors that monitor and control the allocation of resources. In the resource management unit, a first monitor, at a first level, monitors and allocates elements below the first level. A second monitor, at a second level, monitors and allocates elements at the first level. The framework is extendable from the hierarchy of the first and second levels to higher levels where monitors at higher levels each monitor lower level elements in a hierarchical tree. If a failure occurs down the hierarchy, a higher level monitor restarts an element at a lower level. If a failure occurs up the hierarchy, a lower level monitor restarts an element at a higher level. Each of the monitors includes termination code that causes an element to terminate if duplicate elements have been restarted for the same job. The termination code in one embodiment includes suicide code whereby an element will self-destruct when the element detects that it is an unnecessary duplicate element.
Inventors
- 👤 Drew Shaffer Roselli
- 👤 Rico Blaser
- 👤 Mikel Carl Lechner
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-30 15:31
CPC Classifications
G; G06; G06F; G06F11/00; G06F11/07; G06F11/14; G06F11/1402; G06F11/1415; G06F11/1438
Claims Analysis
Total Claims: 74
Claim 1
Independent Claim
93 words
A fault tolerant computer system for executing one or more jobs on one or more nodes, comprising, a hierarchy of monitors for monitoring operations in the computer system including, one or more first monitors for monitoring first operations and, for any particular one of said first operations that fails, for restarting another instance of said particular one of said first operations, one or more second monitors for monitoring said first monitors and, if any particular one of said first monitors fails, for restarting another instance of said particular one of said first monitors.
Claim 2
Dependent Claim
93 words
A fault tolerant computer system for executing one or more jobs on one or more nodes, comprising, a hierarchy of monitors for monitoring operations in the computer system including, one or more first monitors for monitoring first operations and, for any particular one of said first operations that fails, for restarting another instance of said particular one of said first operations, one or more second monitors for monitoring said first monitors and, if any particular one of said first monitors fails, for restarting another instance of said particular one of said first monitors.
Claim 3
Dependent Claim
53 words
The system of claim 1 wherein, said one or more of said second monitors are monitored by at least one of said first monitors and, if any particular one of said second monitors fails, said at least, one of said first monitors restarts another instance of said particular one of said second monitors.
Claim 4
Dependent Claim
35 words
The system of claim 2 wherein one or more of said second monitors operates to commit suicide if more than one of said another instance of said particular one of said second monitors is restarted.
Claim 5
Dependent Claim
24 words
The system of claim 1 wherein, said nodes operate to execute processes in a service unit, a communication unit and a resource management unit.
Claim 6
Dependent Claim
43 words
The system of claim 1 wherein each of said nodes includes a computer having an operating system, wherein pluralities of nodes form clusters and wherein each cluster has a corresponding instantiation of said hierarchy of monitors for monitoring operations in the computer system.
Claim 7
Dependent Claim
103 words
The system of claim 5 wherein each instantiation of said hierarchy of monitors includes, a first instantiation of said one or more first monitors for monitoring first instantiation operations and, for any particular one of said first instantiation operations that fails, for restarting another instance of said particular one of said first instantiation operations, a second instantiation of said one or more second monitors for monitoring said first monitors of said first instantiation and, if any particular one of said first monitors of said first instantiation fails, for restarting another instance of said particular one of said first monitors of said first instantiation.
Claim 8
Dependent Claim
78 words
The system of claim 5 including first and second instantiations and wherein, said one or more of said second monitors of said second instantiation are monitored by at least one of said first monitors of said first instantiation and, if any particular one of said one or more of said second monitors of said second instantiation fails, for restarting another instance of said particular one of said one or more of said second monitors of said second instantiation.
Claim 9
Dependent Claim
41 words
The system of claim 1 wherein, said second monitors maintain a record of particular ones of the first monitors that are active and corresponding active particular ones of said first operations being monitored by said particular ones of the first monitors.
Claim 10
Dependent Claim
59 words
The system of claim 8 wherein, said second monitors use said record to ensure that active particular ones of said first operations monitored by a failing one of said particular ones of the first monitors that are active is monitored by a new instance of said failing one of said particular ones of the first monitors that are active.
Claim 11
Dependent Claim
54 words
The system of claim 1 wherein said hierarchy of monitors includes, one or more additional monitors for monitoring said first monitors or said second monitors, and, if any particular one of said first monitors or said second monitors fails, restarting another instance of said particular one of said first monitors or said second monitors.
Claim 12
Dependent Claim
63 words
The system of claim 10 wherein said hierarchy of monitors includes, one or more other monitors for monitoring said first monitors, said second monitors or said additional monitors, and, if any particular one of said first monitors, said second monitors or said additional monitors fails, restarting another instance of said particular one of said first monitors, said second monitors or said additional monitors.
Claim 13
Dependent Claim
43 words
The system of claim 1 wherein, said first operations are jobs running on said nodes for providing services and, for any particular one of said jobs that fails, one of said first monitors restarts another instance of said particular one of said jobs.
Claim 14
Dependent Claim
12 words
The system of claim 12 wherein said jobs implement e-commerce transaction services.
Claim 15
Dependent Claim
14 words
The system of claim 12 wherein said jobs implement transaction services for financial instruments.
Claim 16
Dependent Claim
37 words
The system of claim 12 wherein said first monitors are host agents for monitoring operations of a plurality of jobs on a plurality of nodes where each job is monitored by only one of said host agents.
Claim 17
Dependent Claim
41 words
The system of claim 12 wherein said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents.
Claim 18
Dependent Claim
70 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and said one or more second monitors includes one or more local coordinators operating on a second level where each local coordinator monitors one or more of said agents.
Claim 19
Dependent Claim
69 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs.
Claim 20
Dependent Claim
81 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs on other of said nodes than said particular one of said nodes.
Claim 21
Dependent Claim
123 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs, said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein a particular one of said local coordinators runs on a particular one of said nodes where an agent monitored by said particular one of said local coordinators runs.
Claim 22
Dependent Claim
125 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs, said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein a particular one of said local coordinators runs on a particular one of said nodes other than where an agent monitored by said particular one of said local coordinators runs.
Claim 23
Dependent Claim
142 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein said hierarchy of monitors includes, one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors, and wherein a particular one of said third monitors that monitors said particular one of said second monitors runs on a different node than a node where said particular one of said second monitors runs.
Claim 24
Dependent Claim
81 words
The system of claim 22 wherein said hierarchy of monitors includes, one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors, and wherein a particular one of said fourth monitors that monitors said particular one of said third monitors runs on a different node than a node where said particular one of said third monitors runs.
Claim 25
Dependent Claim
138 words
The system of claim 12 wherein, said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein said hierarchy of monitors includes, one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors, and wherein a particular one of said third monitors that monitors said particular one of said second monitors runs on a node where said particular one of said second monitors runs.
Claim 26
Dependent Claim
77 words
The system of claim 24 wherein said hierarchy of monitors includes, one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors, and wherein a particular one of said fourth monitors that monitors said particular one of said third monitors runs on a node where said particular one of said third monitors runs.
Claim 27
Dependent Claim
46 words
The system of claim 1 wherein said hierarchy of monitors includes, one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors.
Claim 28
Dependent Claim
34 words
The system of claim 26 wherein one or more of said second monitors operates to commit suicide if more than one of said instance of said particular one of said second monitors is restarted.
Claim 29
Dependent Claim
30 words
The system of claim 26 wherein said one or more third monitors run on different ones of said nodes than ones of said nodes on which said second monitors run.
Claim 30
Dependent Claim
46 words
The system of claim 26 wherein said hierarchy of monitors includes, one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors.
Claim 31
Dependent Claim
30 words
The system of claim 29 wherein said one or more fourth monitors run on different ones of said nodes than ones of said nodes on which said third monitors run.
Claim 32
Dependent Claim
33 words
The system of claim 29 wherein said one or more fourth monitors run on ones of said nodes which are the same as ones of said nodes on which said third monitors run.
Claim 33
Dependent Claim
34 words
The system of claim 29 wherein one or more of said third monitors operates to commit suicide if more than one of said instance of said particular one of said third monitors is restarted.
Claim 34
Dependent Claim
19 words
The system of claim 1 having a resource management unit including a load-balancing for distributing jobs among said nodes.
Claim 35
Dependent Claim
15 words
The system of claim 1 having a resource management unit including a persistent storage unit.
Claim 36
Dependent Claim
14 words
The system of claim 1 having a resource management unit including an interface unit.
Claim 37
Dependent Claim
20 words
The system of claim 1 wherein, each of said nodes includes a plurality of computers each having an operating system.
Claim 38
Dependent Claim
31 words
The system of claim 1 having a plurality of clusters of said nodes, each cluster having a corresponding instantiation of said hierarchy of monitors for monitoring operations in the computer system.
Claim 39
Dependent Claim
29 words
The system of claim 37 wherein, each of said clusters of nodes operates to execute processes organized into a service unit, a communication unit and a resource management unit.
Claim 40
Dependent Claim
23 words
The system of claim 37 wherein, said clusters of nodes are organized into groups, each group having one or more of said clusters.
Claim 41
Dependent Claim
47 words
The system of claim 39 wherein, a first one of said groups is located at a geographic location remote from a second one of said groups and said first one of said groups is connected to said second one of said groups by one or more networks.
Claim 42
Dependent Claim
37 words
The system of claim 39 wherein, a first one of said groups is organized to execute on one subset of data and a second one of said groups is organized to execute on another subset of data.
Claim 43
Dependent Claim
39 words
The system of claim 37 wherein, a first one of said groups is organized to execute on one subset of data and a second one of said groups is organized to provide backup for said one subset of data.
Claim 44
Dependent Claim
72 words
The system of claim 1 wherein, said first operations are jobs running on said nodes for providing services, said first monitor senses one or more conditions that can cause any particular one of said jobs to fail whether or not said particular one of said jobs has actually failed, one of said first monitors terminates said particular one of said jobs and restarts another instance of said particular one of said jobs.
Claim 45
Dependent Claim
43 words
The system of claim 43 wherein, said one of said first monitors that terminates said particular one of said jobs restarts said another instance of said particular one of said jobs in an environment where said one or more conditions are not present.
Claim 46
Dependent Claim
33 words
The system of claim 43 wherein, said one of said conditions is a node failure and said another instance of said particular one of said jobs is started on a different non-failing node.
Claim 47
Dependent Claim
35 words
The system of claim 43 wherein, said one of said conditions is a job failure and said another instance of said particular one of said jobs is started as a new instance of said job.
Claim 48
Dependent Claim
43 words
The system of claim 46 wherein, said another instance of said particular one of said jobs is started as a new instance of said job on a node the same as a node on which said particular one of said jobs was running.
Claim 49
Dependent Claim
43 words
The system of claim 46 wherein, said another instance of said particular one of said jobs is started as a new instance of said job on a new node different from a node on which said particular one of said jobs was running.
Claim 50
Dependent Claim
51 words
The system of claim 1 wherein each of said nodes includes a computer and wherein new ones of said nodes are added to the system without disturbing the operations of other of said nodes in the computer system and wherein jobs are assigned dynamically to said new ones of said nodes.
Claim 51
Dependent Claim
53 words
The system of claim 1 wherein each of said nodes includes a computer and wherein ones of said nodes are removed from the system without disturbing the operations of other of said nodes in the computer system and wherein particular jobs are reassigned dynamically to other of said nodes in the computer system.
Claim 52
Dependent Claim
79 words
The system of claim 1 wherein each of said nodes includes a computer of one type and wherein new ones of said nodes are added to the system including upgraded computers of a different type without disturbing the operations of other of said nodes in the computer system and wherein jobs are assigned dynamically from said other of said nodes to said new ones of said nodes to provide dynamic upgrade of said system without stopping said particular jobs.
Claim 53
Dependent Claim
44 words
The system of claim 1 wherein pluralities of nodes form clusters and wherein particular ones of said clusters are assigned for processing particular jobs at particulars times and wherein other ones of said clusters are assigned for processing said particular jobs at other times.
Claim 54
Dependent Claim
16 words
The system of claim 52 wherein said particular times and said other times are follow-the-sun times.
Claim 55
Dependent Claim
17 words
The system of claim 1 wherein a delay time is controlled before the restart of a job.
Claim 56
Dependent Claim
30 words
The system of claim 1 having an interface that allows humans to monitor the health of the system and to log statistics about uptime of each component in the system.
Claim 57
Dependent Claim
24 words
The system of claim 1 wherein a delay time is applied before said restarting another instance of said particular one of said first operations.
Claim 58
Dependent Claim
80 words
The system of claim 1 wherein in said hierarchy of monitors, said one or more of said second monitors are monitored by at least one of said first monitors and, if any particular one of said second monitors fails, said at least one of said first monitors, after a first delay time, restarts another instance of said particular one of said second monitors on a node other than a node on which said particular one of said second monitors failed.
Claim 59
Dependent Claim
43 words
The system of claim 57 wherein, if more than one instance of said another instance of said particular one of said second monitors is restarted, all but one instance of said another instance of said particular one of said second monitors commits suicide.
Claim 60
Dependent Claim
59 words
The system of claim 57 wherein said hierarchy of monitors includes, one or more additional monitors for monitoring said first monitors and said second monitors, and, if any particular one of said first monitors or said second monitors fails, restarting, after a second delay time, another instance of said particular one of said first monitors or said second monitors.
Claim 61
Dependent Claim
54 words
The system of claim 59 wherein, if more than one of instance of said another instance of said particular one of said first monitors or said second monitors is restarted, all but one instance of said another instance of said particular one of said first monitors or said second monitors operates to commit suicide.
Claim 62
Dependent Claim
68 words
The system of claim 59 wherein said hierarchy of monitors includes, one or more other monitors for monitoring said first monitors, said second monitors and said additional monitors, and, if any particular one of said first monitors, said second monitors or said additional monitors fails, restarting, after a third delay time, another instance of said particular one of said first monitors, said second monitors or said additional monitors.
Claim 63
Dependent Claim
59 words
The system of claim 61 wherein, if more than one instance of said another instance of said particular one of said first monitors, said second monitors or said additional monitors is restarted, all but one instance of said another instance of said particular one of said first monitors, said second monitors or said additional monitors operates to commit suicide.
Claim 64
Dependent Claim
58 words
The system of claim 1 wherein, said first operations are jobs running on said nodes for providing services where a particular first one of said jobs associated with a first customer is running on a particular first node and a particular second one of said jobs associated with a second customer is running on said particular first node.
Claim 65
Dependent Claim
70 words
The system of claim 1 wherein, said first operations are jobs running on said nodes for providing services where a particular first one of said jobs associated with a first customer is running on a particular first node and a particular second one of said jobs associated with a second customer is running on a particular second node whereby said first customer job is isolated from said second customer job.
Claim 66
Dependent Claim
111 words
The system of claim 1 wherein, said first operations are jobs running on said nodes for providing services where, particular first ones of said jobs are associated with a first customer with one of said particular first ones of said jobs running on a particular first node and with another one of said particular first ones of said jobs running on a particular other node; particular second ones of said jobs are associated with a second customer with one of said particular second ones of said jobs running on a particular second node and with another one of said particular second ones of said jobs running on said particular other node.
Claim 67
Dependent Claim
25 words
The system of claim 1 including transaction initiators for starting said first operations as one or more jobs to initiate a transaction in a service.
Claim 68
Dependent Claim
25 words
The system of claim 1 including transaction processors for starting said first operations as one or more jobs to process a transaction in a service.
Claim 69
Dependent Claim
63 words
The system of claim 1 including, transaction initiators for starting first ones or more of said first operations as one or more first jobs on a first node to initiate a transaction in a service; transaction processors for starting other ones or more of said first operations as one or more other jobs on another node to process said transaction in said service.
Claim 70
Dependent Claim
64 words
The system of claim 1 including, transaction initiators for starting first ones or more of said first operations as one or more first jobs on a first node to initiate a transaction in a service; transaction processors for starting other ones or more of said first operations as one or more other jobs on said first node to process said transaction in said service.
Claim 71
Dependent Claim
99 words
In a fault tolerant computer system operating to execute one or more jobs on one or more nodes where the computer system includes a hierarchy of monitors for monitoring operations in the computer system, the method comprising, monitoring first operations with one or more first monitors and, for any particular one of said first operations that fails, restarting another instance of said particular one of said first operations, monitoring said first monitors with one or more second monitors and, if any particular one of said first monitors fails, restarting another instance of said particular one of said first monitors.
Claim 72
Dependent Claim
99 words
In a fault tolerant computer system operating to execute one or more jobs on one or more nodes where the computer system includes a hierarchy of monitors for monitoring operations in the computer system, the method comprising, monitoring first operations with one or more first monitors and, for any particular one of said first operations that fails, restarting another instance of said particular one of said first operations, monitoring said first monitors with one or more second monitors and, if any particular one of said first monitors fails, restarting another instance of said particular one of said first monitors.
Claim 73
Dependent Claim
53 words
The method of claim 70 wherein, monitoring said one or more of said second monitors with at least one of said first monitors and, if any particular one of said second monitors fails, restarting with said at least one of said first monitors another instance of said particular one of said second monitors.
Claim 74
Dependent Claim
35 words
The method of claim 71 wherein one or more of said second monitors operates to commit suicide if more than one of said another instance of said particular one of said second monitors is restarted.
References Cited (Backward Citations)
US5740357A
Cited
US5748882A
Cited
US5638510A
Cited
US5805785A
Cited
US6477663B1
Cited
US6266781B1
Cited
US6460149B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-04-06 | 2007-10-06 | 2008-04-06 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-04-06 | 2011-10-06 | 2012-04-06 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-04-06 | 2015-10-06 | 2016-04-06 | $7700.00 | Window Closed / Lapsed |