Techniques for DLM optimization with re-mastering events

Claim 3

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the one or more terminating nodes are associated with a plurality of terminating hash value ranges in the system hash map; and the range replacement set is selected from a subset of said plurality of terminating hash value ranges.

Claim 4

The method of claim 2 , wherein: the method further includes the step of setting a tunable system parameter value indicating a maximum amount for resource usage of resources to re-master in one re-mastering event; the replacement range set has value for resource usage that is less than or equal to the tunable system parameter value.

Claim 5

The method of claim 3 , wherein the resource usage is indicative of a number of open resources.

Claim 6

The method of claim 3 , wherein the resource usage is indicative of a number of open locks held on the resources.

Claim 7

The method of claim 1 , wherein: data gathered during the time interval includes data indicating one or more terminating nodes left the cluster during the time interval; and no re-mastering event is performed because the re-mastering event condition was not satisfied.

Claim 8

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include at least one hash value range is associated with a terminating node which left the cluster during a prior time interval, and said at least one hash value range has not been re-mastered in a prior re-mastering event; and the step of re-mapping includes selecting the at least one hash value range for the replacement range set.

Claim 9

The method of claim 7 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating a joining node joined the cluster; and the step of re-mapping includes replacing data that maps the at least one hash value range to the terminating node with data that maps the at least one hash value range to the joining node.

Claim 10

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the one or more terminating nodes are associated with a plurality of terminating hash value ranges in the system hash map; the method further includes the step of setting a tunable system parameter value indicating a minimum amount for resource usage of resources to re-master in one re-mastering event; the one or more factors include data indicating that the plurality of terminating hash value ranges has a value for resource usage that is greater than or equal to the tunable system parameter value.

Claim 11

The method of claim 9 , wherein the resource usage is indicative of a number of open resources.

Claim 12

The method of claim 9 , wherein the resource usage is indicative of a number of open locks held on the resources.

Claim 13

The method of claim 7 , wherein the one or more factors include data indicating the terminating node has not rejoined the cluster.

Claim 14

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating a joining node joined the cluster; and the step of re-mapping includes inserting data that maps at least one hash value range in the replacement range set to the joining node.

Claim 15

The method of claim 13 , the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set does not include an original hash value range still associated in the system hash map with an original node with which the original hash value range was associated in the system hash map during said initializing step.

Claim 16

The method of claim 13 , the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set does not include an in-use hash value range encompasing a hash value of a resource with an open lock.

Claim 17

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating one or more joining nodes; the method further includes the step of setting a tunable system parameter value indicating a maximum amount for resource usage of resources to re-master in one re-mastering event; and the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set has a value for resource usage that is less than or equal to the tunable system parameter value.

Claim 18

The method of claim 1 , wherein the re-mastering event condition is not satisfied unless a minimum wait time has elapsed since a prior re-mastering event.

Claim 19

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the step of performing the re-mastering event further comprises closing all locks owned by the one or more terminating nodes.

Claim 20

The method of claim 1 , wherein: data gathered during the time interval includes data indicating whether an abnormally terminating node left the cluster abnormally during the time interval; the one or more factors include data indicating an abnormally terminating node; and the step of performing the re-mastering event includes freezing lock requests only for dubious resources associated with one or more dubious hash value ranges associated with the abnormally terminating node, and selecting only the one or more dubious hash value ranges for the replacement range set.

Claim 21

The method of claim 19 , wherein: the method further comprises maintaining a plurality of shadow resource lock objects, each object residing on a shadow node of the cluster, the object associated with a particular resource mastered on a node different than the shadow node, the object including lock information for all requests for the particular resource from processes on the shadow node; and the step of performing the re-mastering event further includes transferring lock information to the new set from certain shadow objects of the plurality of shadow resource lock objects, the certain shadow objects associated with dubious resources mastered by the abnormally terminating node.

Claim 22

The method of claim 20 , wherein the step of performing the re-mastering event further includes, if the system recovers the dubious resources, marking the dubious resources as valid and unfreezing lock requests for the dubious resources.

Claim 25

The method of claim 22 , wherein: the weight is related to a proportion of the plurality of hash value ranges desirably mastered at each node, a negative weight for a particular node indicating none of the plurality of hash value ranges is desirably mastered at the particular node; data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; said performing the re-mastering event includes, before re-mapping, re-setting the negative weight of the particular node to a non-negative weight, if the particular node is a standby node for the one or more terminating nodes.

Claim 26

The method of claim 22 , wherein: the weight is related to a first proportion of the plurality of hash value ranges desirably mastered at each node; and the one or more factors include data indicating no terminating node is identified during the time interval, data indicating no joining node is identified during the time interval, and a second proportion of hash value ranges associated with each master node in the hash map differs substantially from a third proportion responsive to the weight of each node.

Claim 27

The method of claim 22 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating one or more joining nodes; the method further includes the step of setting a tunable system parameter value indicating a maximum number of hashed ranges to re-master in one re-mastering event; and the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that a number of hash value ranges in the replacement range set is less than or equal to the tunable system parameter value.

Claim 28

The method of claim 22 , wherein: the weight is related to a first proportion of the plurality of hash value ranges desirably mastered at each node; and the step of re-mapping includes selecting a particular hash value range for the replacement range set if the particular range set is associated with a particular master node in the hash map, the particular node having actual proportion of hash value ranges associated with the particular master node in the hash map that is greater than a desirable proportion responsive to the weight of the particular node.

Claim 29

The method of claim 26 , wherein the step of re-mapping includes selecting a second node for the new set of master nodes if the second node has a second actual proportion of hash value ranges associated with the second node in the hash map that is less than a second desirable proportion responsive to the weight of the second node.

Claim 30

A computer-readable medium bearing instructions for distributing over a cluster of one or more active nodes, management of locks on shared resources, the instructions causing one or more processors to perform: initializing a system hash map that establishes a mapping between a plurality of hash value ranges and one or more master nodes; initially distributing management of said locks based on mapping; monitoring the cluster to gather data during a time interval, said data including data that identifies how much resource usage is made of resources hashed to each hash value range of the plurality of hash value ranges; determining whether a re-mastering event condition is satisfied based on one or more factors, said one or more factors including said resource usage; and if the re-mastering event condition is satisfied, performing a re-mastering event including re-mapping said system hash map by replacing data that maps a replacement range set to an old set of corresponding master nodes with data that maps said replacement range set to a new set of corresponding master nodes from the one or more active nodes, wherein the replacement range set includes one or more hash value ranges of the plurality of hash value ranges, and transferring lock information from the old set of one or more master nodes to the new set of one or more master nodes.

Claim 31

A computer-readable medium bearing instructions for distributing over a cluster of one or more active nodes, management of locks on shared resources, the instructions causing one or more processors to perform: initializing a system hash map that establishes a mapping between a plurality of hash value ranges and one or more master nodes; initially distributing management of said locks based on mapping; monitoring the cluster to gather data during a time interval, said data including data that identifies how much resource usage is made of resources hashed to each hash value range of the plurality of hash value ranges; determining whether a re-mastering event condition is satisfied based on one or more factors, said one or more factors including said resource usage; and if the re-mastering event condition is satisfied, performing a re-mastering event including re-mapping said system hash map by replacing data that maps a replacement range set to an old set of corresponding master nodes with data that maps said replacement range set to a new set of corresponding master nodes from the one or more active nodes, wherein the replacement range set includes one or more hash value ranges of the plurality of hash value ranges, and transferring lock information from the old set of one or more master nodes to the new set of one or more master nodes.

Claim 32

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the one or more terminating nodes are associated with a plurality of terminating hash value ranges in the system hash map; and the range replacement set is selected from a subset of said plurality of terminating hash value ranges.

Claim 33

The computer-readable medium of claim 29 , wherein: the instructions causing the one or more processors to further perform setting a tunable system parameter value indicating a maximum amount for resource usage of resources to re-master in one re-mastering event; the replacement range set has value for resource usage that is less than or equal to the tunable system parameter value.

Claim 34

The computer-readable medium of claim 30 , wherein the resource usage is indicative of a number of open resources.

Claim 35

The computer-readable medium of claim 30 , wherein the resource usage is indicative of a number of open locks held on the resources.

Claim 36

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating one or more terminating nodes left the cluster during the time interval; and no re-mastering event is performed because the re-mastering event condition was not satisfied.

Claim 37

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include at least one hash value range is associated with a terminating node which left the cluster during a prior time interval, and said at least one hash value range has not been re-mastered in a prior re-mastering event; and the step of re-mapping includes selecting the at least one hash value range for the replacement range set.

Claim 38

The computer-readable medium of claim 34 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating a joining node joined the cluster; and the step of re-mapping includes replacing data that maps the at least one hash value range to the terminating node with data that maps the at least one hash value range to the joining node.

Claim 39

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the one or more terminating nodes are associated with a plurality of terminating hash value ranges in the system hash map; the computer-readable medium further includes the step of setting a tunable system parameter value indicating a minimum amount for resource usage of resources to re-master in one re-mastering event; the one or more factors include data indicating that the plurality of terminating hash value ranges has a value for resource usage that is greater than or equal to the tunable system parameter value.

Claim 40

The computer-readable medium of claim 36 , wherein the resource usage is indicative of a number of open resources.

Claim 41

The computer-readable medium of claim 36 , wherein the resource usage is indicative of a number of open locks held on the resources.

Claim 42

The computer-readable medium of claim 34 , wherein the one or more factors include data indicating the terminating node has not rejoined the cluster.

Claim 43

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating a joining node joined the cluster; and the step of re-mapping includes inserting data that maps at least one hash value range in the replacement range set to the joining node.

Claim 44

The computer-readable medium of claim 40 , the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set does not include an original hash value range still associated in the system hash map with an original node with which the original hash value range was associated in the system hash map during said initializing step.

Claim 45

The computer-readable medium of claim 40 , the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set does not include an in-use hash value range encompasing a hash value of a resource with an open lock.

Claim 46

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating one or more joining nodes; the instructions cause the one or more processors to further perform setting a tunable system parameter value indicating a maximum amount for resource usage of resources to re-master in one re-mastering event; and the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that the replacement range set has a value for resource usage that is less than or equal to the tunable system parameter value.

Claim 47

The computer-readable medium of claim 28 , wherein the re-mastering event condition is not satisfied unless a minimum wait time has elapsed since a prior re-mastering event.

Claim 48

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; the step of performing the re-mastering event further comprises closing all locks owned by the one or more terminating nodes.

Claim 49

The computer-readable medium of claim 28 , wherein: data gathered during the time interval includes data indicating whether an abnormally terminating node left the cluster abnormally during the time interval; the one or more factors include data indicating an abnormally terminating node; and the step of performing the re-mastering event includes freezing lock requests only for dubious resources associated with one or more dubious hash value ranges associated with the abnormally terminating node, and selecting only the one or more dubious hash value ranges for the replacement range set.

Claim 50

The computer-readable medium of claim 46 , wherein: the instructions cause the one or more processors to further perform maintaining a plurality of shadow resource lock objects, each object residing on a shadow node of the cluster, the object associated with a particular resource mastered on a node different than the shadow node, the object including lock information for all requests for the particular resource from processes on the shadow node; and the step of performing the re-mastering event further includes transferring lock information to the new set from certain shadow objects of the plurality of shadow resource lock objects, the certain shadow objects associated with dubious resources mastered by the abnormally terminating node.

Claim 51

The computer-readable medium of claim 47 , wherein the step of performing the re-mastering event further includes, if the system recovers the dubious resources, marking the dubious resources as valid and unfreezing lock requests for the dubious resources.

Claim 52

A computer-readable medium bearing instructions for distributing over a cluster of one or more active nodes, management of locks on shared resources, the instructions causing one or more processors to perform: setting a weight for each node that may be included in the cluster; initializing a system hash map that establishes a mapping between a plurality of hash value ranges and one or more master nodes; initially distributing management of said locks based on mapping; monitoring the cluster to gather data during a time interval, said data including data that identifies a number of hash value ranges of the plurality of hash value ranges that are mapped to each master node of the one or more master nodes during the time interval; determining whether a re-mastering event condition is satisfied based on one or more factors, said one or more factors including a first weight associated with a first active node of the cluster and the number of hash value ranges mapped to the first active node; and if the re-mastering event condition is satisfied, performing a re-mastering event including re-mapping said system hash map by replacing data that maps a replacement range set to an old set of corresponding master nodes with data that maps said replacement range set to a new set of corresponding master nodes from the one or more active nodes, wherein the replacement range set includes one or more hash value ranges of the plurality of hash value ranges, and transferring lock information from the old set of one or more master nodes to the new set of one or more master nodes.

Claim 53

A computer-readable medium bearing instructions for distributing over a cluster of one or more active nodes, management of locks on shared resources, the instructions causing one or more processors to perform: setting a weight for each node that may be included in the cluster; initializing a system hash map that establishes a mapping between a plurality of hash value ranges and one or more master nodes; initially distributing management of said locks based on mapping; monitoring the cluster to gather data during a time interval, said data including data that identifies a number of hash value ranges of the plurality of hash value ranges that are mapped to each master node of the one or more master nodes during the time interval; determining whether a re-mastering event condition is satisfied based on one or more factors, said one or more factors including a first weight associated with a first active node of the cluster and the number of hash value ranges mapped to the first active node; and if the re-mastering event condition is satisfied, performing a re-mastering event including re-mapping said system hash map by replacing data that maps a replacement range set to an old set of corresponding master nodes with data that maps said replacement range set to a new set of corresponding master nodes from the one or more active nodes, wherein the replacement range set includes one or more hash value ranges of the plurality of hash value ranges, and transferring lock information from the old set of one or more master nodes to the new set of one or more master nodes.

Claim 54

The computer-readable medium of claim 49 , wherein: the weight is related to a proportion of the plurality of hash value ranges desirably mastered at each node, a negative weight for a particular node indicating none of the plurality of hash value ranges is desirably mastered at the particular node; data gathered during the time interval includes data indicating whether a terminating node left the cluster during the time interval; the one or more factors include data indicating one or more terminating nodes; said performing the re-mastering event includes, before re-mapping, re-setting the negative weight of the particular node to a non-negative weight, if the particular node is a standby node for the one or more terminating nodes.

Claim 55

The computer-readable medium of claim 49 , wherein: the weight is related to a first proportion of the plurality of hash value ranges desirably mastered at each node; and the one or more factors include data indicating no terminating node is identified during the time interval, data indicating no joining node is identified during the time interval, and a second proportion of hash value ranges associated with each master node in the hash map differs substantially from a third proportion responsive to the weight of each node.

Claim 56

The computer-readable medium of claim 49 , wherein: data gathered during the time interval includes data indicating whether a joining node joined the cluster during the time interval; the one or more factors include data indicating one or more joining nodes; the instructions cause the one or more processors to further perform setting a tunable system parameter value indicating a maximum number of hashed ranges to re-master in one re-mastering event; and the step of re-mapping includes selecting one or more hash value ranges for the replacement range set so that a number of hash value ranges in the replacement range set is less than or equal to the tunable system parameter value.

Claim 57

The computer-readable medium of claim 49 , wherein: the weight is related to a first proportion of the plurality of hash value ranges desirably mastered at each node; and the step of re-mapping includes selecting a particular hash value range for the replacement range set if the particular range set is associated with a particular master node in the hash map, the particular node having actual proportion of hash value ranges associated with the particular master node in the hash map that is greater than a desirable proportion responsive to the weight of the particular node.

Claim 58

The computer-readable medium of claim 53 , wherein the step of re-mapping includes selecting a second node for the new set of master nodes if the second node has a second actual proportion of hash value ranges associated with the second node in the hash map that is less than a second desirable proportion responsive to the weight of the second node.