US5666486A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Multiprocessor cluster membership manager framework
Assignee: EMC Corp
•
Filing: 1995-06-23
•
Granted: 1997-09-09
•
Expires: 2015-06-23
Abstract
Abstract A shared-disk cluster system includes a cluster membership manager framework which coordinates the joining or leaving among all nodes in a cluster including taking the multiple layers of involved subsystems through transitions. Subsystems are notified of transitions in particular order depending upon the transition, and all nodes' subsystems receiving a notification must process that notification prior to another layer of subsystems being notified. One of the subsystems registered for notification is an event manager in user space. The event manager carries out transfers of client services, including user applications, resulting from nodes joining and leaving the cluster. This includes a registration and launch service which registers a node, or multiple nodes, in a cluster which claims, or is assigned, responsibility for the service and provides an optional launching function which initiates the client service upon successful registration.
Inventors
- 👤 Robert A. Alfieri
- 👤 James T. Compton
- 👤 Andrew R. Huber
- 👤 Paul T. McGrath
- 👤 Khaled S. Soufi
- 👤 Brian J. Thorstad
- 👤 Eric R. Vook
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-29 15:38
CPC Classifications
G; G06; G06F; G06F9/00; G06F9/06; G06F9/46; G06F9/50; G06F9/5061; G06F11/00; G06F11/07; G06F11/14; G06F11/1402; G06F11/1415; G06F11/142; G06F11/1425; G06F11/1479; G06F11/1482; H; H04; H04L; H04L41/00; H04L41/08; H04L41/0893; G06F11/16; G06F11/20; G06F11/202; G06F11/2035; G06F11/2046; G06F2209/00; G06F2209/50; G06F2209/505
Claims Analysis
Total Claims: 53
Claim 1
Independent Claim
93 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing said shared resource, said method including notifying all levels of subsystems running on nodes presently in the cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster.
Claim 2
Dependent Claim
93 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing said shared resource, said method including notifying all levels of subsystems running on nodes presently in the cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster.
Claim 3
Dependent Claim
24 words
The method of claim 1 wherein the subsystems are interdependent in levels, with higher level subsystems dependent on the operation of lower level subsystems.
Claim 4
Dependent Claim
51 words
The method of claim 2 including notifying one of said subsystems on all of said nodes presently in the cluster of a transition, processing that notification at said one of said subsystems prior to notifying another of said subsystems on all of said nodes presently in the cluster of the transition.
Claim 5
Dependent Claim
31 words
The method of claim 3 including notifying subsystems beginning with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a node joining the cluster.
Claim 6
Dependent Claim
32 words
The method of claim 3 including notifying subsystems beginning with higher lever subsystems and proceeding in sequence through lower levels of subsystems of a transition of a node gracefully leaving the cluster.
Claim 7
Dependent Claim
35 words
The method of claim 3 including notifying subsystems beginning with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a node forced from the cluster by other processors.
Claim 8
Dependent Claim
18 words
The method of claim 2 wherein said subsystems include a higher level subsystem which interacts with user programs.
Claim 9
Dependent Claim
33 words
The method of claim 7 wherein said higher level subsystem includes a service which automatically and atomically transfers user programs to other nodes when the node executing the user programs leaves the cluster.
Claim 10
Dependent Claim
24 words
The method of claim 2 wherein said subsystems include a distributed lock manager subsystem, a virtual disk manager subsystem and a shared file subsystem.
Claim 11
Dependent Claim
55 words
The method of claim 1 wherein for a transition of a node joining the cluster, said method includes the steps of: a) registering subsystems of the joining node to receive transition notifications; b) joining the node to the cluster; and c) notifying registered subsystems in the cluster that the joining node has joined the cluster.
Claim 12
Dependent Claim
70 words
The method of claim 1 wherein for a transition of one node being forced out of the cluster by another node, said method includes the steps of: a) the another node notifying registered subsystems that the one node is being forced out of the cluster; and b) transferring registered programs executing on said one node atomically to a different node and recovering the programs to execute on said different node.
Claim 13
Dependent Claim
113 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of initiating client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, said method including choosing a node for each client service, registering the client service with that node, and notifying nodes presently in the cluster that the particular client service is registered with the particular node, whereby the particular service can be transferred to another node if the particular node leaves the cluster.
Claim 14
Dependent Claim
113 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of initiating client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, said method including choosing a node for each client service, registering the client service with that node, and notifying nodes presently in the cluster that the particular client service is registered with the particular node, whereby the particular service can be transferred to another node if the particular node leaves the cluster.
Claim 15
Dependent Claim
34 words
The method of claim 12 further including launching a client service on a node according to an action parameter included with the client service in response to registering that client service with that node.
Claim 16
Dependent Claim
46 words
The method of claim 13 further including grouping client services as a parent client service and at least one child client service, registering grouped client services with the same node and launching grouped client services according to an action parameter included with the parent client service.
Claim 17
Dependent Claim
44 words
The method of claim 12 wherein said choosing a node includes providing a database of choosing factors for the client service and applying said choosing factors to the nodes presently in the cluster, said choosing factors establishing rules relating nodes to the client service.
Claim 18
Dependent Claim
22 words
The method of claim 15, wherein said choosing factors are selected from the group including allowable nodes, disallowable nodes and node preferences.
Claim 19
Dependent Claim
23 words
The method of claim 12 further including notifying nodes joining the cluster that the particular client service is registered with the particular nodes.
Claim 20
Dependent Claim
21 words
The method of claim 12 including monitoring a client service registered with a node at that node using an event watcher.
Claim 21
Dependent Claim
29 words
The method of claim 18 including enabling the event watcher in response to registering the client service and disabling the event watcher in response to de-registering the client service.
Claim 22
Dependent Claim
40 words
The method of claim 12 wherein said multiple subsystems includes a cluster membership manager which controls which of said nodes are presently in the cluster and wherein said cluster membership manager provides notification of which nodes are in the cluster.
Claim 23
Dependent Claim
25 words
The method of claim 20 wherein said initiating services on a particular one of said nodes is performed by another one of said multiple subsystems.
Claim 24
Dependent Claim
164 words
A multiprocessor cluster system having multiple nodes, a shared resource accessible to all nodes, a cluster communication medium between said nodes, and multiple subsystems on each of said nodes, comprising: a cluster membership manager subsystem adapted to notify subsystems running on nodes presently in the cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster; an event manager subsystem adapted to detect and react to cluster errors; and a registration and launch service responsive to said event manager and adapted to initiate client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch subsystem chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 25
Dependent Claim
164 words
A multiprocessor cluster system having multiple nodes, a shared resource accessible to all nodes, a cluster communication medium between said nodes, and multiple subsystems on each of said nodes, comprising: a cluster membership manager subsystem adapted to notify subsystems running on nodes presently in the cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster; an event manager subsystem adapted to detect and react to cluster errors; and a registration and launch service responsive to said event manager and adapted to initiate client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch subsystem chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 26
Dependent Claim
26 words
The multiprocessor cluster system in claim 22 wherein the subsystems are interdependent in levels, with higher level subsystems dependent on the operation of lower level subsystems.
Claim 27
Dependent Claim
64 words
The multiprocessor cluster system in claim 23 wherein said membership manager subsystem notifies one of said subsystems on all of said nodes presently in the cluster of a transition, and that one of said subsystems on all nodes processes that notification prior to said membership manager subsystem notifying another of said subsystems on all of said nodes presently in the cluster of the transition.
Claim 28
Dependent Claim
36 words
The multiprocessor cluster system in claim 24 wherein said membership manager notifies subsystems beginning with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a node joining the cluster.
Claim 29
Dependent Claim
37 words
The multiprocessor cluster system in claim 24 wherein said membership manager notifies subsystems beginning with higher level subsystems and proceeding in sequence through lower levels of subsystems of a transition of a node gracefully leaving the cluster.
Claim 30
Dependent Claim
40 words
The multiprocessor cluster system in claim 24 wherein said membership manager notifies subsystems beginning with lower level subsystems and proceeding in sequence through higher levels of subsystems of a transition of a node forced from the cluster by other processors.
Claim 31
Dependent Claim
40 words
The multiprocessor cluster system in claim 22 wherein said registration and launch service launches a client service on a node according to an action parameter included with the client service in response to registering that client service with that node.
Claim 32
Dependent Claim
63 words
The multiprocessor cluster system in claim 28 wherein said registration and launch service is adapted to group client services as a parent client service and at least one child client service, and wherein said registration and launch service further registers grouped client services with the same node and launches grouped client services according to an action parameter included with the parent client service.
Claim 33
Dependent Claim
55 words
The multiprocessor cluster system in claim 22 wherein said registration and launch service includes a database of choosing factors for the client service and applies said choosing factors to the nodes presently in the cluster to choose the node for registering a client service, said choosing factors establishing rules relating nodes to the client service.
Claim 34
Dependent Claim
30 words
The multiprocessor cluster system in claim 22 wherein said registration and launch service further notifies nodes joining the cluster that the particular client service is registered with the particular node.
Claim 35
Dependent Claim
27 words
The multiprocessor cluster system in claim 22 wherein said event manager includes an event watcher for monitoring a client service registered with a node at that node.
Claim 36
Dependent Claim
124 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and including multiple interdependent levels of subsystems, one of said subsystems being a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to notify all levels of subsystems running on nodes presently in a cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster.
Claim 37
Dependent Claim
124 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and including multiple interdependent levels of subsystems, one of said subsystems being a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to notify all levels of subsystems running on nodes presently in a cluster of transitions of nodes joining and leaving the cluster in order to provide a consistent view of active membership in the cluster.
Claim 38
Dependent Claim
125 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and including a registration and launch service which defines a computer process for allocating said nodes to perform client services, said registration and launch service adapted to initiate client services on particular ones of nodes so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch service chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 39
Dependent Claim
125 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, a shared resource accessible to all processors, and including a registration and launch service which defines a computer process for allocating said nodes to perform client services, said registration and launch service adapted to initiate client services on particular ones of nodes so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch service chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 40
Dependent Claim
236 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, and a shared resource accessible to all nodes, comprising: multiple subsystems that are interdependent in levels, with higher level subsystems dependent on the operation of lower level subsystems; one of said subsystems comprising a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to notifying subsystems running on nodes presently in a cluster of transitions of processors joining and leaving the cluster in order to provide a consistent view of active membership in the cluster; and one of said subsystems including a registration and launch service which defines a computer process for allocating said nodes to perform client services, said registration and launch service adapted to initiate client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch service chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 41
Dependent Claim
236 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, and a shared resource accessible to all nodes, comprising: multiple subsystems that are interdependent in levels, with higher level subsystems dependent on the operation of lower level subsystems; one of said subsystems comprising a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to notifying subsystems running on nodes presently in a cluster of transitions of processors joining and leaving the cluster in order to provide a consistent view of active membership in the cluster; and one of said subsystems including a registration and launch service which defines a computer process for allocating said nodes to perform client services, said registration and launch service adapted to initiate client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, wherein said registration and launch service chooses a node for each client service, registers the client service with that node, and notifies nodes presently in the cluster that the particular service is registered with the particular node.
Claim 42
Dependent Claim
119 words
In a multiprocessor system having multiple nodes, and a shared resource accessible to all nodes, a method of initiating client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, said method including providing a registration and launch service which defines a computer process, said registration and launch service adapted to selecting one of said nodes for a client service, registering that client service with the selected node and launching that client service on the selected node according to an action parameter included with that client service in response to registering that client service with the selected node.
Claim 43
Dependent Claim
119 words
In a multiprocessor system having multiple nodes, and a shared resource accessible to all nodes, a method of initiating client services on particular ones of said nodes in a cluster so that it appears to said client services that the entire cluster is substantially a single node containing said shared resource, said method including providing a registration and launch service which defines a computer process, said registration and launch service adapted to selecting one of said nodes for a client service, registering that client service with the selected node and launching that client service on the selected node according to an action parameter included with that client service in response to registering that client service with the selected node.
Claim 44
Dependent Claim
19 words
The method in claim 36 including transferring the client service to another node if that node leaves the cluster.
Claim 45
Dependent Claim
22 words
The method in claim 36 wherein said transferring includes relaunching the client service on the another node according to said action parameter.
Claim 46
Dependent Claim
104 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said method including taking said levels of said subsystems through node transitions in a manner that a particular level of said subsystems is taken through a node transition at all said ones of said nodes prior to taking a different level of said subsystems through the node transition.
Claim 47
Dependent Claim
104 words
In a multiprocessor system having multiple nodes, a shared resource accessible to all nodes and multiple interdependent levels of subsystems on each of said nodes, a method of combining particular ones of said nodes in a cluster so that it appears to users that the entire cluster is substantially a single node containing the shared resource, said method including taking said levels of said subsystems through node transitions in a manner that a particular level of said subsystems is taken through a node transition at all said ones of said nodes prior to taking a different level of said subsystems through the node transition.
Claim 48
Dependent Claim
24 words
The method of claim 39 wherein a higher level of said subsystems is dependent on the operation of a lower level of said subsystems.
Claim 49
Dependent Claim
40 words
The method of claim 40 including taking a lower level of said subsystems through a node transition prior to taking a higher level of said subsystems through the node transition if the node transition is a node joining the cluster.
Claim 50
Dependent Claim
46 words
The method of claim 40 including taking a lower level of said subsystems through a node transition prior to taking a higher level of said subsystems through the node transition if the node transition is a node ungracefully being forced from the cluster by other nodes.
Claim 51
Dependent Claim
41 words
The method of claim 40 including taking a higher level of said subsystems through a node transition prior to taking a lower level of said subsystems through the node transition if the node transition is a node gracefully leaving the cluster.
Claim 52
Dependent Claim
135 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, and a shared resource accessible to all nodes, and including multiple interdependent levels of subsystems; one of said subsystems being a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster for that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to taking all levels of subsystems through node transitions in a manner that a particular level of said subsystems is taken through a node transition at all said ones of said nodes prior to taking a different level of said subsystems through the node transition.
Claim 53
Dependent Claim
135 words
A computer usable medium in which program code is embodied, said program code defining an operating system for a multiprocessor cluster system having multiple nodes, and a shared resource accessible to all nodes, and including multiple interdependent levels of subsystems; one of said subsystems being a cluster membership manager subsystem which defines a computer process for combining particular ones of said nodes in a cluster for that it appears to users that the entire cluster is substantially a single node containing the shared resource, said cluster membership manager subsystem adapted to taking all levels of subsystems through node transitions in a manner that a particular level of said subsystems is taken through a node transition at all said ones of said nodes prior to taking a different level of said subsystems through the node transition.
References Cited (Backward Citations)
US4849877A
Cited
US5202971A
Cited
US4980857A
Cited
US5390316A
Cited
US5446841A
Cited
US5448470A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2000-09-08 | 2001-03-10 | 2001-09-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2004-09-08 | 2005-03-10 | 2005-09-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2008-09-08 | 2009-03-10 | 2009-09-09 | $7700.00 | Window Closed / Lapsed |