CN109067649B - Node processing method and device, storage medium and electronic equipment - Google Patents
Node processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109067649B CN109067649B CN201810858537.XA CN201810858537A CN109067649B CN 109067649 B CN109067649 B CN 109067649B CN 201810858537 A CN201810858537 A CN 201810858537A CN 109067649 B CN109067649 B CN 109067649B
- Authority
- CN
- China
- Prior art keywords
- node
- routing
- pressure
- cluster
- data slot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/70—Routing based on monitoring results
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
Abstract
The invention discloses a node processing method and device, a storage medium and electronic equipment, and relates to the technical field of data storage. The node processing method and the node processing device are applied to an inflixdb cluster, wherein the inflixdb cluster comprises a plurality of configured nodes and a newly added node, and the node processing method comprises the following steps: acquiring route statistical information of each data slot of each configured node in a cluster; determining the routing pressure of each configured node according to the routing statistical information of each data slot; and transferring the data slots of the configured nodes in the cluster to the newly added nodes according to the routing pressure of each configured node. The method and the device can solve the problem that the storage pressure of the nodes in the existing infiluxdb cluster is large and the service requirements cannot be met.
Description
Technical Field
The present disclosure relates to the field of data storage technologies, and in particular, to a node processing method, a node processing apparatus, a storage medium, and an electronic device.
Background
The time sequence database is more and more favored by developers due to the advantages of ordered time, intuition, convenience in distinguishing and the like. As one of the timing databases which are currently most concerned, infiluxdb has been widely applied to a plurality of application scenarios such as log monitoring.
The inflixdb cluster may not match the actual storage service due to network, node hardware equipment, service content, etc. For example, as the amount of traffic increases, the storage capacity of the existing infilxdb cluster cannot meet the storage demand, and in this case, adding a new storage node in the infixtb cluster has become a better solution to the storage bottleneck. At present, no scheme for adding nodes in the infiluxdb cluster exists.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure aims to provide a node processing method, a node processing apparatus, a storage medium, and an electronic device, so as to solve the problem of large storage pressure of nodes in an existing infiluxdb cluster at least to a certain extent.
According to an aspect of the present disclosure, there is provided a node processing method applied to an inflixdb cluster including a plurality of configured nodes and a newly added node, the node processing method including: acquiring route statistical information of each data slot of each configured node in a cluster; determining the routing pressure of each configured node according to the routing statistical information of each data slot; and transferring the data slots of the configured nodes in the cluster to the newly added nodes according to the routing pressure of each configured node.
In an exemplary embodiment of the present disclosure, the routing statistics of each data slot in the cluster are persisted into a configuration file every predetermined time; the obtaining of the route statistical information of each data slot of each configured node in the cluster includes: and acquiring the routing statistical information of each data slot of each configured node in the cluster from the configuration file.
In an exemplary embodiment of the present disclosure, the routing pressure is a number of times that all data slots on a node are routed within a preset time period.
In an exemplary embodiment of the present disclosure, transferring the data slots of the configured nodes in the cluster to the new node according to the routing pressure of each configured node includes: calculating the sum of the routing pressure of n configured nodes in the cluster; wherein n is the total number of nodes of configured nodes in the cluster; taking the ratio of the routing pressure sum to n +1 as an average routing pressure; and transferring the data slot of the configured node in the cluster to the newly added node according to the average routing pressure, so that the routing pressure of the newly added node is the average routing pressure.
In an exemplary embodiment of the disclosure, transferring the data slots of the configured nodes in the cluster to the newly added node according to the average routing pressure comprises: determining a node with the routing pressure greater than the average routing pressure in the configured nodes as a node to be transferred; and transferring the data slot of the node to be transferred to the newly added node.
In an exemplary embodiment of the present disclosure, transferring the data slot of the node to be transferred to the newly added node includes: and transferring the data slot of the node to be transferred to the newly added node according to the proportion of the routing pressure of the node to be transferred.
In an exemplary embodiment of the present disclosure, transferring the data slot of the node to be transferred to the newly added node includes: determining the pressure excess amount of the routing pressure of each node to be transferred exceeding the average routing pressure; calculating the proportion of the pressure excess of each node to be transferred; and transferring the data slot of the node to be transferred to the newly added node according to the proportion of the pressure excess amount.
According to an aspect of the present disclosure, a node processing apparatus is provided, which is applied to an inflixdb cluster including a plurality of configured nodes and a newly added node, and includes a routing information obtaining module, a routing pressure determining module, and a data slot transferring module.
Specifically, the routing information obtaining module may be configured to obtain routing statistical information of each data slot of each configured node in the cluster; the routing pressure determining module may be configured to determine routing pressure of each configured node according to the routing statistical information of each data slot; the data slot transferring module may be configured to transfer the data slots of the configured nodes in the cluster to the newly added node according to the routing pressure of each configured node.
In an exemplary embodiment of the present disclosure, the routing statistics of each data slot in the cluster are persisted into a configuration file every predetermined time; the routing information obtaining module may include a routing information obtaining unit.
Specifically, the route information obtaining unit may be configured to obtain, from the configuration file, route statistical information of each data slot of each configured node in the cluster.
In an exemplary embodiment of the present disclosure, the routing pressure is a number of times that all data slots on a node are routed within a preset time period.
In an exemplary embodiment of the present disclosure, the data slot transfer module may include a sum of pressure calculation unit, an average pressure calculation unit, and a data slot transfer unit.
Specifically, the pressure sum calculating unit may be configured to calculate a routing pressure sum of n configured nodes in the cluster; wherein n is the total number of nodes of configured nodes in the cluster; the average pressure calculation unit may be configured to use a ratio of the sum of the routing pressures to n +1 as an average routing pressure; the data slot transferring unit may be configured to transfer the data slot of the configured node in the cluster to the newly added node according to the average routing pressure, so that the routing pressure of the newly added node is the average routing pressure.
In an exemplary embodiment of the present disclosure, the data slot transferring unit may include a to-be-transferred node determining unit and a to-be-transferred node data slot transferring unit.
Specifically, the to-be-transferred node determining unit may be configured to determine, as the to-be-transferred node, a node of the configured nodes whose routing pressure is greater than the average routing pressure; the data slot transferring unit of the node to be transferred can be used for transferring the data slot of the node to be transferred to the newly added node.
In an exemplary embodiment of the present disclosure, the node-to-be-transferred data slot transfer unit may include a first data slot transfer subunit.
Specifically, the first data slot transferring subunit may be configured to transfer the data slot of the node to be transferred to the newly added node according to a ratio of the routing pressure of the node to be transferred.
In an exemplary embodiment of the present disclosure, the to-be-transferred node data slot transfer unit may include an excess amount determination subunit, an excess proportion calculation subunit, and a second data slot transfer subunit.
Specifically, the excess amount determining subunit may be configured to determine a pressure excess amount by which the routing pressure of each node to be transferred exceeds the average routing pressure; the excess proportion calculation subunit can be used for calculating the proportion of the pressure excess amount of each node to be transferred; the second data slot transfer subunit may be configured to transfer the data slot of the node to be transferred to the newly added node according to a proportion of the pressure excess.
According to an aspect of the present disclosure, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the node processing method of any one of the above.
According to an aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above node processing methods via execution of the executable instructions.
In the technical solutions provided by some embodiments of the present disclosure, by determining the routing pressure of each configured node according to the routing statistical information of each data slot of each configured node in the cluster, and transferring the data slot of the configured node in the cluster to a newly added node according to the routing pressure of each configured node, on one hand, adding a node in an infiluxdb cluster can be realized to meet the requirement of increasing the service storage amount; on the other hand, the scheme realizes the transfer of the data slot based on the routing pressure of the configured nodes, and can better realize load balance.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:
FIG. 1 schematically illustrates a flow chart of a node processing method according to an exemplary embodiment of the present disclosure;
FIG. 2 schematically illustrates a block diagram of a node processing apparatus according to an exemplary embodiment of the present disclosure;
FIG. 3 schematically illustrates a block diagram of a routing information acquisition module according to an exemplary embodiment of the present disclosure;
FIG. 4 schematically illustrates a block diagram of a data slot transfer module, according to an exemplary embodiment of the present disclosure;
FIG. 5 schematically illustrates a block diagram of a data slot transfer unit, according to an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates a block diagram of a node data slot transfer unit to be transferred, according to an exemplary embodiment of the present disclosure;
FIG. 7 schematically illustrates a block diagram of a node-to-be-transferred data slot transfer unit, according to another exemplary embodiment of the present disclosure;
FIG. 8 shows a schematic diagram of a storage medium according to an example embodiment of the present disclosure; and
fig. 9 schematically shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the steps. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
Before describing the node processing method and apparatus of the exemplary embodiments of the present disclosure, a description will be given of a cluster fragmentation scheme of infiluxdb.
Specifically, first, the client may assign a unique identifier to each piece of data to be written; then, after the routing node obtains the data, hash processing may be performed on the unique identifier to obtain a corresponding hash value, and the hash value is divided by a preset value (e.g., 16384) and then is left to determine a data slot (slot) corresponding to each piece of data; and then, sending each piece of data to the influxdb node corresponding to the data slot. The corresponding relationship between the data slots and the inflixdb nodes is configured in advance, for example, 16384 data slots may be respectively corresponding to two inflixdb nodes, where the first inflixdb node stores the 1 st data slot to the 8192 th data slot, and the second inflixdb node stores the 8193 th data slot to the 16384 th data slot.
However, the inflixdb cluster may not match the actual storage traffic due to network, node hardware equipment, traffic content, etc. For example, as traffic increases, the storage capacity of existing infiluxdb clusters has been unable to meet the storage demand.
In view of this, the present disclosure provides a node processing method and apparatus, so as to better solve the above problem.
Fig. 1 schematically shows a flowchart of a node processing method of an exemplary embodiment of the present disclosure. The node processing method disclosed by the invention can be applied to an inflixdb cluster which comprises a plurality of configured nodes and a newly added node. Referring to fig. 1, the node processing method may include the steps of:
s12, obtaining the routing statistical information of each data slot of each configured node in the cluster.
It should be understood that a cluster may be composed of a plurality of nodes, each node may include one or more data slots, and when a server writes data, the server may write the data into a certain data slot of a certain node of the cluster, and the specific writing process may refer to the above-described fragmentation scheme. In addition, when a data needs to be read, the server may read the data from the data slot of the node in the cluster.
The routing of the data slot may mean the process of reading data from or writing data to the data slot at a time. The routing statistics described in this disclosure may include the routing frequency of the data slot, i.e., the number of routes per unit time. In addition, the route statistical information may further include information such as the total route times and route time of the data slot, which is not limited in this disclosure.
In an exemplary embodiment of the disclosure, when a data slot is routed, a server may obtain routing information of the data slot and perform statistics to obtain routing statistics information. In addition, the routing statistics of each data slot in the cluster may be persisted into a configuration file at predetermined time intervals, where the predetermined time may be set based on actual traffic needs, for example, may be 8 hours, 24 hours, and the like. The configuration file may be a storage unit existing in the server, and may be a file configured in other forms, and the present disclosure does not make any particular limitation on the specific format and storage location of the configuration file.
Specifically, the obtaining, by the server, the routing statistical information of each data slot of each configured node in the cluster may include obtaining, from the configuration file, the routing statistical information of each data slot of each configured node in the cluster.
S14, determining the routing pressure of each configured node according to the routing statistical information of each data slot.
In an exemplary embodiment of the present disclosure, the routing pressure may refer to the number of times all data slots on a node are routed within a preset time period, which may be, for example, 8 hours, 12 hours, 24 hours, and the like. In other examples of the present disclosure, the routing pressure may also be determined synthetically using statistical characteristics of routing statistics for all data slots on the node. The statistical characteristics may include, but are not limited to, an average, a standard deviation, etc. of the total number of routes.
It can be seen that the routing pressure described in this disclosure can be shown in numerical form.
And S16, transferring the data slots of the configured nodes in the cluster to the newly added nodes according to the routing pressure of each configured node.
In an exemplary embodiment of the present disclosure, first, the server may calculate a sum of routing pressures of n configured nodes in the cluster; and n is the total number of the configured nodes in the cluster.
The ratio of the sum of the routing pressures to n +1 may then be taken as the average routing pressure. That is, the sum of the routing pressures of all nodes in the cluster can be divided into n +1 parts, where n +1 means that after a new node is added, the total number of nodes in the cluster is n + 1.
For example, there are 3 configured nodes in the cluster, which are respectively a node a, a node B, and a node C, and the routing pressures of the nodes are 12000, 4000, and 8000, respectively, and in the case of the newly added node D, the average routing pressure may be calculated, that is: (12000+4000+8000)/4 ═ 6000.
Then, the data slots of the configured nodes in the cluster can be transferred to the newly added nodes according to the average routing pressure, so that the routing pressure of the newly added nodes is the average routing pressure.
Specifically, a node with a routing pressure greater than the average routing pressure among configured nodes in the cluster may be determined as a node of the data slot to be transferred (denoted as a node to be transferred). Still taking the above as an example, because the routing pressures of node a and node C are both greater than the average routing pressure 6000, in this case, node a and node C are the nodes to be transferred.
After the node to be transferred is determined, the data slot of the node to be transferred can be transferred to the newly added node. The following two embodiments will specifically describe the process of transferring the data slot to the newly added node.
Embodiment one of data slot transfer：
The data slot of the node to be transferred can be transferred to the newly added node according to the proportion of the routing pressure of the node to be transferred. Still referring to the above example, the ratio of the routing pressures of node A and node C is 3:2, so that 60% (i.e., 3600) of the routing pressure 6000 of the newly added node D can come from node A, and the remaining 40% (i.e., 2400) can come from node C.
Specifically, on one hand, starting from the first data slot of the node A, accumulating the routing times of the first data slot until the accumulated routing times of the data slot reaches 3600, and transferring the part of data slot to the newly added node D; on the other hand, starting from the first data slot of the node C, the routing times are accumulated, and when the accumulated routing times of the data slot reaches 2400, the part of the data slot is transferred to the newly added node D.
After the transfer is completed, the routing pressures of the node A, the node B, the node C and the node D in the cluster are 8400, 4000, 5600 and 6000 respectively.
Example two of data Slot transfer：
First, a pressure overshoot by which the routing pressure of each node to be transferred exceeds the average routing pressure may be determined. Also as an example of the above data configuration, node a has a pressure overshoot of 6000 and node C has a pressure overshoot of 2000.
Subsequently, the ratio of the pressure excess of each node to be transferred can be calculated. For example, the ratio of the pressure overshoot at node A to node C is 3: 1.
And then, the data slot of the node to be transferred can be transferred to the newly added node according to the proportion of the pressure excess amount. For example, 75% (i.e., 4500) of the routing pressure 6000 of the newly added node D may come from node a, and the remaining 25% (i.e., 1500) may come from node C.
After the transfer is completed, the routing pressures of the node A, the node B, the node C and the node D in the cluster are 7500, 4000, 6500 and 6000 respectively.
It can be seen that the load balancing effect in the second embodiment is better than that in the first embodiment.
The present disclosure only transfers data slots for configured nodes that are above the average routing pressure, rather than directly achieving an absolute average of the routing pressures of the nodes. The absolute averaging scheme necessarily consumes a large amount of system computing resources as well as transmission capabilities. The scheme of the present disclosure can relatively reduce system resource consumption while better realizing load balancing.
According to the node processing method disclosed by the invention, on one hand, the node can be added in the infiluxdb cluster to meet the requirement of increasing the service memory space; on the other hand, the scheme realizes the transfer of the data slot based on the routing pressure of the configured nodes, and can better realize load balance.
It should be noted that although the various steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that these steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
Further, the present exemplary embodiment also provides a node processing apparatus.
Fig. 2 schematically shows a block diagram of a node processing apparatus of an exemplary embodiment of the present disclosure. The node processing apparatus may be applied to an infiluxdb cluster comprising a plurality of configured nodes and a newly added node. Referring to fig. 2, the node processing apparatus 2 according to an exemplary embodiment of the present disclosure may include a routing information acquisition module 21, a routing pressure determination module 23, and a data slot transfer module 25.
Specifically, the routing information obtaining module 21 may be configured to obtain routing statistical information of each data slot of each configured node in the cluster; the routing pressure determining module 23 may be configured to determine the routing pressure of each configured node according to the routing statistical information of each data slot; the data slot transferring module 25 may be configured to transfer the data slots of the configured nodes in the cluster to the new node according to the routing pressure of each configured node.
According to the node processing device disclosed by the invention, on one hand, the node can be added in the infiluxdb cluster to meet the requirement of increasing the service storage capacity; on the other hand, the scheme realizes the transfer of the data slot based on the routing pressure of the configured nodes, and can better realize load balance.
According to the exemplary embodiment of the present disclosure, the route statistical information of each data slot in the cluster is persisted into a configuration file every predetermined time; wherein, referring to fig. 3, the routing information obtaining module 21 may include a routing information obtaining unit 301.
Specifically, the routing information obtaining unit 301 may be configured to obtain, from the configuration file, routing statistical information of each data slot of each configured node in the cluster.
According to an exemplary embodiment of the present disclosure, the routing pressure is the number of times that all data slots on a node are routed within a preset time period.
According to an exemplary embodiment of the present disclosure, referring to fig. 4, the data slot transfer module 25 may include a sum of pressure calculation unit 401, an average pressure calculation unit 403, and a data slot transfer unit 405.
Specifically, the pressure sum calculating unit 401 may be configured to calculate a routing pressure sum of n configured nodes in the cluster; wherein n is the total number of nodes of configured nodes in the cluster; the average pressure calculation unit 403 may be configured to use a ratio of the routing pressure sum to n +1 as an average routing pressure; the data slot transferring unit 405 may be configured to transfer the data slot of the configured node in the cluster to the newly added node according to the average routing pressure, so that the routing pressure of the newly added node is the average routing pressure.
According to an exemplary embodiment of the present disclosure, referring to fig. 5, the data slot transfer unit 405 may include a to-be-transferred node determination unit 501 and a to-be-transferred node data slot transfer unit 503.
Specifically, the to-be-transferred node determining unit 501 may be configured to determine, as the to-be-transferred node, a node with a routing pressure greater than the average routing pressure in the configured nodes; the data slot transferring unit 503 of the node to be transferred may be configured to transfer the data slot of the node to be transferred to the new node.
According to an exemplary embodiment of the present disclosure, referring to fig. 6, the to-be-transferred node data slot transfer ticket 503 may include a first data slot transfer subunit 6001.
Specifically, the first data slot transferring subunit 6001 may be configured to transfer the data slot of the node to be transferred to the newly added node according to a ratio of the routing pressure of the node to be transferred.
According to an exemplary embodiment of the present disclosure, referring to fig. 7, a node-to-be-transferred data slot transfer unit 701 of another embodiment of the present disclosure may include an excess amount determination subunit 7001, an excess ratio calculation subunit 7003, and a second data slot transfer subunit 7005.
Specifically, the excess amount determination subunit 7001 may be configured to determine a pressure excess amount by which the routing pressure of each node to be transferred exceeds the average routing pressure; the excess proportion calculation subunit 7003 can be used for calculating the proportion of the pressure excess amount of each node to be transferred; a second data slot transfer subunit 7005 may be used to transfer the data slot of the node to be transferred to the newly added node according to a ratio of pressure overshoot.
Since each functional module of the program operation performance analysis apparatus according to the embodiment of the present invention is the same as that in the embodiment of the present invention, it is not described herein again.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
Referring to fig. 8, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 900 according to this embodiment of the invention is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is only an example and should not bring any limitations to the function and scope of use of the embodiments of the present invention.
As shown in fig. 9, the electronic device 900 is embodied in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.
Wherein the storage unit stores program code that is executable by the processing unit 910 to cause the processing unit 910 to perform steps according to various exemplary embodiments of the present invention described in the above section "exemplary methods" of the present specification. For example, the processing unit 910 may perform steps S12 through S16 as shown in fig. 1.
The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM)9201 and/or a cache memory unit 9202, and may further include a read only memory unit (ROM) 9203.
The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 900, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 900 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 950. Also, the electronic device 900 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 960. As shown, the network adapter 960 communicates with the other modules of the electronic device 900 via the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.
Claims (6)
1. A node processing method is applied to an inflixdb cluster, and the inflixdb cluster comprises a plurality of configured nodes and a newly added node, and is characterized by comprising the following steps:
acquiring route statistical information of each data slot of each configured node in a cluster;
determining the routing pressure of each configured node according to the routing statistical information of each data slot;
calculating the sum of the routing pressure of n configured nodes in the cluster; wherein n is the total number of nodes of configured nodes in the cluster; taking the ratio of the routing pressure sum to n +1 as an average routing pressure; determining a node with the routing pressure greater than the average routing pressure in the configured nodes as a node to be transferred; determining the pressure excess amount of the routing pressure of each node to be transferred exceeding the average routing pressure; calculating the proportion of the pressure excess of each node to be transferred; and transferring the data slot of the node to be transferred to the newly added node according to the proportion of the pressure excess amount.
2. The node processing method according to claim 1, wherein the routing statistics of the data slots in the cluster are persisted into a configuration file every predetermined time; the obtaining of the route statistical information of each data slot of each configured node in the cluster includes:
and acquiring the routing statistical information of each data slot of each configured node in the cluster from the configuration file.
3. The node processing method according to claim 1, wherein the routing pressure is a number of times all data slots on the node are routed within a preset time period.
4. A node processing apparatus applied to an inflixdb cluster including a plurality of configured nodes and a newly added node, the node processing apparatus comprising:
the routing information acquisition module is used for acquiring routing statistical information of each data slot of each configured node in the cluster;
a routing pressure determining module, configured to determine routing pressure of each configured node according to the routing statistical information of each data slot;
the data slot transfer module is used for calculating the sum of the routing pressures of n configured nodes in the cluster; wherein n is the total number of nodes of configured nodes in the cluster; taking the ratio of the routing pressure sum to n +1 as an average routing pressure; determining a node with the routing pressure greater than the average routing pressure in the configured nodes as a node to be transferred; determining the pressure excess amount of the routing pressure of each node to be transferred exceeding the average routing pressure; calculating the proportion of the pressure excess of each node to be transferred; and transferring the data slot of the node to be transferred to the newly added node according to the proportion of the pressure excess amount.
5. A storage medium on which a computer program is stored, the computer program, when executed by a processor, implementing the node processing method of any one of claims 1 to 3.
6. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the node processing method of any of claims 1 to 3 via execution of the executable instructions.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810858537.XA CN109067649B (en) | 2018-07-31 | 2018-07-31 | Node processing method and device, storage medium and electronic equipment |
| PCT/CN2018/106272 WO2020024392A1 (en) | 2018-07-31 | 2018-09-18 | Node processing method and apparatus, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810858537.XA CN109067649B (en) | 2018-07-31 | 2018-07-31 | Node processing method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109067649A CN109067649A (en) | 2018-12-21 |
| CN109067649B true CN109067649B (en) | 2021-06-22 |
Family
ID=64832037
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810858537.XA Active CN109067649B (en) | 2018-07-31 | 2018-07-31 | Node processing method and device, storage medium and electronic equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109067649B (en) |
| WO (1) | WO2020024392A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111475535B (en) * | 2020-03-09 | 2024-02-06 | 咪咕文化科技有限公司 | Data storage and access method and device |
| CN117372166B (en) * | 2023-10-26 | 2024-03-08 | 北京开科唯识技术股份有限公司 | Efficient tail-end distribution processing method, device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653630A (en) * | 2015-12-25 | 2016-06-08 | 北京奇虎科技有限公司 | Data migration method and apparatus for distributed database |
| CN106844510A (en) * | 2016-12-28 | 2017-06-13 | 北京五八信息技术有限公司 | The data migration method and device of a kind of distributed experiment ＆ measurement system |
| CN106973091A (en) * | 2017-03-23 | 2017-07-21 | 中国工商银行股份有限公司 | Distributed memory fast resampling method and system, main control server |
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN107665246A (en) * | 2017-09-13 | 2018-02-06 | 北京知道未来信息技术有限公司 | Dynamic date migration method and chart database cluster based on chart database |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9805108B2 (en) * | 2010-12-23 | 2017-10-31 | Mongodb, Inc. | Large distributed database clustering systems and methods |
| US9703610B2 (en) * | 2011-05-16 | 2017-07-11 | Oracle International Corporation | Extensible centralized dynamic resource distribution in a clustered data grid |
| CN102810116B (en) * | 2012-06-29 | 2015-01-07 | 安科智慧城市技术（中国）有限公司 | Automatic routing and load balancing method and system based on database connection |
| US9514168B2 (en) * | 2013-03-15 | 2016-12-06 | Sunil Sayyaparaju | Method and system of implementing a database index in shared memory |
| CN105893205A (en) * | 2015-11-20 | 2016-08-24 | 乐视云计算有限公司 | Method and system for monitoring containers created based on docker |
| CN106527981B (en) * | 2016-10-31 | 2020-04-28 | 华中科技大学 | Data fragmentation method of self-adaptive distributed storage system based on configuration |
-
2018
- 2018-07-31 CN CN201810858537.XA patent/CN109067649B/en active Active
- 2018-09-18 WO PCT/CN2018/106272 patent/WO2020024392A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653630A (en) * | 2015-12-25 | 2016-06-08 | 北京奇虎科技有限公司 | Data migration method and apparatus for distributed database |
| CN106844510A (en) * | 2016-12-28 | 2017-06-13 | 北京五八信息技术有限公司 | The data migration method and device of a kind of distributed experiment ＆ measurement system |
| CN106973091A (en) * | 2017-03-23 | 2017-07-21 | 中国工商银行股份有限公司 | Distributed memory fast resampling method and system, main control server |
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN107665246A (en) * | 2017-09-13 | 2018-02-06 | 北京知道未来信息技术有限公司 | Dynamic date migration method and chart database cluster based on chart database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109067649A (en) | 2018-12-21 |
| WO2020024392A1 (en) | 2020-02-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111090628A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN103970520A (en) | Resource management method and device in MapReduce framework and framework system with device | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN110019498B (en) | Log synchronization method and device, storage medium and electronic equipment | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| CN111600957A (en) | File transmission method, device and system and electronic equipment | |
| CN111985831A (en) | Scheduling method and device of cloud computing resources, computer equipment and storage medium | |
| CN109067649B (en) | Node processing method and device, storage medium and electronic equipment | |
| CN111752916B (en) | Data acquisition method and device, computer readable storage medium and electronic equipment | |
| CN112817687A (en) | Data synchronization method and device | |
| CN115016890A (en) | Virtual machine resource allocation method and device, electronic equipment and storage medium | |
| CN113076175B (en) | Memory sharing method and device for virtual machine | |
| WO2019232932A1 (en) | Node processing method and apparatus, and computer-readable storage medium and electronic device | |
| CN111159237B (en) | System data distribution method and device, storage medium and electronic equipment | |
| CN114090201A (en) | Resource scheduling method, device, equipment and storage medium | |
| CN114217790A (en) | Interface scheduling method and device, electronic equipment and medium | |
| CN109586970B (en) | Resource allocation method, device and system | |
| CN110247802B (en) | Resource configuration method and device for cloud service single-machine environment | |
| CN113760469A (en) | Distributed computing method and device | |
| CN113760920A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN113760940A (en) | Quota management method, device, equipment and medium applied to distributed system | |
| CN112364268A (en) | Resource acquisition method and device, electronic equipment and storage medium | |
| CN110677463B (en) | Parallel data transmission method, device, medium and electronic equipment | |
| CN111046430B (en) | Data processing method and device, storage medium and electronic equipment | |
| US10884937B2 (en) | Reference cache maintenance optimizer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |