CN115878568A - Data updating method, device, equipment and computer readable storage medium - Google Patents

Abstract

The embodiment of the application provides a data updating method, a device, equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring update data to be written into a target cluster; acquiring state information of each node in a target cluster; determining a plurality of idle nodes from each node according to the state information of each node; and creating a write thread for each idle node, and writing the update data to the idle node through the write thread. According to the data updating method provided by the embodiment of the application, the state information of each node is used as an evaluation standard, so that all nodes with updating tasks are idle nodes in a relatively idle state identified by the state information during updating, and the data updating efficiency is improved.

Description

Data updating method, device, equipment and computer readable storage medium

Technical Field

The embodiment of the application relates to the technical field of data updating, in particular to a data updating method, a data updating device, data updating equipment and a computer readable storage medium.

Background

Many document data are stored in nodes of a cluster, and an update script is required to be periodically executed to write updated data.

At present, the cluster data is updated usually by manually judging the peak time of the system according to experience, then setting the idle time, and uploading data on the set nodes, so as to reduce the pressure on the production environment and improve the data updating efficiency. However, the production environment is complex and variable, the peak period of the system is not completely predictable, the time period is taken as a single evaluation criterion, the robustness is not enough, and the system can still be in the peak period during updating, so that the data updating efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data updating method, a data updating device, data updating equipment and a computer readable storage medium, and aims to solve the problem of low updating efficiency in the existing data updating process.

In one aspect, an embodiment of the present application provides a data updating method, including:

acquiring update data to be written into a target cluster;

acquiring state information of each node in a target cluster;

determining a plurality of idle nodes from each node according to the state information of each node;

and creating a write thread for each idle node, and writing the update data into the idle node through the write thread.

On the other hand, an embodiment of the present application further provides a data updating apparatus, including:

the first acquisition module is used for acquiring the update data to be written into the target cluster;

the second acquisition module is used for acquiring the state information of each node in the target cluster;

the node determining module is used for determining a plurality of idle nodes from each node according to the state information of each node;

and the writing module is used for creating a writing thread aiming at each idle node and writing the updating data into the idle node through the writing thread to acquire the state information of each node in the target cluster to be updated.

On the other hand, an embodiment of the present application further provides a data updating apparatus, where the data updating apparatus includes a processor, a memory, and a data updating program stored in the memory and capable of running on the processor, and the processor executes the data updating program to implement the steps in the data updating method.

On the other hand, the embodiment of the present application further provides a computer-readable storage medium, on which a data updating program is stored, where the data updating program is executed by a processor to implement the steps in the data updating method.

In the embodiment of the application, the state information of each node in the target cluster is obtained, then the idle node is determined according to the state information of each node, and finally the update data to be written into the cluster is written into the idle node through the created write-in thread. Compared with the prior art that the time period is used as a single updating evaluation standard, the data updating method provided by the application takes the state information of each node as the evaluation standard, so that all nodes with updating tasks are idle nodes in a relatively idle state identified by the state information during updating, and the data updating efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic view of a scenario of a data updating method provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart diagram illustrating a first embodiment of a data updating method provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart diagram illustrating a second embodiment of a data updating method provided in the embodiments of the present application;

FIG. 4 is a schematic flow chart diagram illustrating a third embodiment of a data updating method provided in the embodiments of the present application;

FIG. 5 is a schematic flow chart diagram illustrating a fourth embodiment of a data updating method provided in the embodiments of the present application;

FIG. 6 is a schematic flow chart of a fifth embodiment of a data updating method provided in the embodiments of the present application

Fig. 7 is a schematic flowchart of a sixth embodiment in a data updating method provided in an embodiment of the present application;

fig. 8 is a schematic flowchart of a seventh embodiment in a data updating method provided in an embodiment of the present application;

fig. 9 is a schematic structural diagram of an embodiment of a data updating apparatus provided in the embodiment of the present application.

Fig. 10 is a schematic structural diagram of an embodiment of the data updating apparatus provided in the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any inventive step, are within the scope of the present invention.

In the embodiments of the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the invention. In the following description, details are set forth for the purpose of explanation. It will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and processes are not shown in detail to avoid obscuring the description of the invention with unnecessary detail. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in the embodiments herein.

Embodiments of the present application provide a data updating method, apparatus, device, and computer-readable storage medium, which are described in detail below.

As shown in fig. 1, fig. 1 is a schematic view of a data updating method in an embodiment of the present application, where the data updating scenario in the embodiment of the present application includes a target cluster 100 and a data updating apparatus 200. The target cluster may be understood as a server storing data, and in general, the target cluster is composed of a plurality of nodes 101, that is, a plurality of individual servers. The data update apparatus 200 generally refers to a terminal apparatus running a script, and the terminal apparatus may be a client that is understood as a cluster.

The data updating apparatus 200 in the embodiment of the present invention is mainly used for: acquiring update data to be written into a target cluster; acquiring state information of each node in a target cluster; determining a plurality of idle nodes from each node according to the state information of each node; and creating a write thread for each idle node, and writing the update data to the idle node through the write thread.

Based on the scene schematic diagram of the data updating method, an embodiment of the data updating method is provided.

As shown in fig. 2, fig. 2 is a schematic flow chart of a first embodiment of a data updating method provided in the embodiment of the present application, where the data updating method in this embodiment includes steps 201 to 204:

and 201, acquiring the updating data to be written into the target cluster.

In this embodiment of the present application, a target cluster refers to a cluster that needs to be updated with data, and update data refers to data that needs to be updated and written into the target cluster. Generally, the present application does not limit the specific form of the cluster and the data, but for convenience of description, the following description will take the example of writing the index data into the ElasticSearch cluster (a kind of search server).

Specifically, writing data into the cluster is implemented by a script, and the script is usually run on a certain terminal device, that is, a client of the target cluster.

It should be noted that, when the data updating apparatus starts the script to write data into the corresponding cluster, the script reads the data to be updated into the memory first, that is, obtains the updated data to be written into the target cluster.

As an alternative of the present application, a scheme is provided in which a data updating apparatus starts to acquire update data to be written into a target cluster to implement subsequent data updating, and specifically, please refer to subsequent fig. 8 and the content explained in the following description.

202, state information of each node in the target cluster is obtained.

In the embodiment of the present application, taking the elastic search cluster as an example, in a normal case, the elastic search cluster may monitor the state of each node during operation, and provides an interface/_ nodes/stages/. That is, the data updating apparatus can query the state statistics of all nodes in the cluster by calling the interface. Specifically, the state statistical information mainly includes current operation information, such as memory utilization, http connection number (i.e., request connection number), CPU occupancy, active thread number, and the like, and of course, the state information may also include some static information, such as storage space, storage space utilization, cache information, file system content, OS information, and the like.

In the embodiment of the present application, it is considered that the state information of the node is mainly used to determine the idle node, that is, the acquired state information needs to be related to the operating state of the node. That is to say, the data updating device does not need to read all the state statistical information of each node of the cluster, but only needs to obtain the current operation information that can be used for describing the operation state of the node, for example, the aforementioned memory utilization rate, the number of requested connections, the CPU occupancy rate, the number of active threads, and the like. Specifically, the state information at least includes one of request connection number, CPU occupancy, memory occupancy, and active thread number.

Of course, it should be noted that the embodiment of the present application describes the status information acquired by the data updating apparatus and specific contents of the status information by taking the ElasticSearch cluster as an example only. For different clusters, the specific means by which the data updating apparatus obtains the state information of each node in the cluster will be different, but all of them are conventional technical means in the art. This application is not described in detail herein. Similarly, for different clusters, the state information that can be acquired by the data updating apparatus may also be different, but in most cases, the number of request connections, the CPU occupancy, the memory occupancy, and the number of active threads are all the state information that can be acquired. Of course, the state information may not be limited to what is shown above, and other information that may describe the operation state of the node may be considered as the state information in the embodiment of the present application.

As an alternative of the present application, a solution is provided in which a data updating apparatus starts to acquire state information of each node in a target cluster to implement subsequent data updating, and specifically, please refer to subsequent fig. 7 and the content explained in the following description.

And 203, determining a plurality of idle nodes from each node according to the state information of each node.

In the embodiment of the present application, it can be known from the foregoing description that the state information acquired by the data updating apparatus describes the operating state of the node, and therefore, whether the node is in the idle state can be determined according to the operating state of the node. Thereby determining a node in an idle state as an idle node.

Specifically, the following contents of fig. 6 and its explanation may be referred to in a manner of determining a plurality of idle nodes from the nodes of the target cluster by presetting some idle conditions and determining whether the state information satisfies the air conditioning condition, specifically according to the state information of each node.

204, for each idle node, creating a write thread and writing update data to the idle node through the write thread.

In the embodiment of the application, for each idle node, the data updating device creates and starts a corresponding write thread, and writes the update data into the corresponding idle node through the write thread.

When the data update apparatus writes the update data to the idle node by the write thread, the data update apparatus may further determine the write speed or the amount of data to be written based on the information of the idle node. Specifically, as an alternative of the present application, considering that a bulk api for batch insert operation is provided in an ElasticSearch cluster, when the api is used, multiple operations of adding and deleting can be completed in a batch through one request operation, that is, the data amount written in a batch at each time can be further determined according to the information of the idle node, and a specific implementation manner please refer to subsequent fig. 3 and the content explained in the following description. As another alternative of the present application, when the data updating apparatus writes the update data into the idle node through the write thread, it is not necessary to write all the update data into each idle node, and only part of the update data may be written, that is, before the update data is written into the idle node, the data writing amount of each idle node may also be set according to the storage occupancy of the idle node, and the specific implementation manner refers to the content of the following fig. 5 and the explanation thereof.

In the embodiment of the application, the state information of each node in the target cluster is obtained, then the idle node is determined according to the state information of each node, and finally the update data to be written into the cluster is written into the idle node through the created write-in thread. Compared with the prior art that the time period is used as a single updating evaluation standard, the data updating method provided by the application takes the state information of each node as the evaluation standard, so that all nodes with updating tasks are idle nodes in a relatively idle state identified by the state information during updating, and the data updating efficiency is improved.

As shown in fig. 3, fig. 3 is a schematic flow chart of a second embodiment of the data updating method provided in the embodiment of the present application.

In the embodiment of the present application, an implementation manner is provided for writing update data to idle nodes in batch through a target write thread, and specifically includes steps 301 to 303:

301, determining the data volume which can be written in batch of the idle node according to the state information of the idle node.

In the embodiment of the present application, it is easily understood that, since the bulk api for the bulk insert operation is provided in the ElasticSearch cluster, data may be written to the idle node in bulk by the write thread. For a certain node, as the amount of data written in a batch is larger, the load on the node is larger. That is, the more idle a certain idle node is, the larger the amount of data that can be written in batch at this time is. Therefore, after the idle nodes in the cluster are screened out according to the state information of the nodes, the batch writable data size of the idle nodes can be further determined according to the state information of the idle nodes.

Specifically, when the status information only includes single information, for example, one of the request connection number, the CPU occupancy, the memory occupancy, and the active thread number, at this time, the mapping relationship between the status information and the batch writable data volume may be directly stored in a preset database, so that the batch writable data volume corresponding to the status information is obtained by querying the database. When the state information includes multiple types, for example, multiple types among the request connection number, the CPU occupancy, the memory occupancy, and the active thread number, at this time, the mapping relationship between the state information and the data amount that can be written in batch is complex, and if the mapping relationship is directly stored in the database one by one, a large amount of storage space is occupied. Specifically, please refer to the following fig. 4 and the explanation thereof.

302, a target write thread corresponding to the data size is created using the data size as a write parameter of the write thread.

In the embodiment of the application, the target write-in thread can be obtained by configuring the write-in parameters corresponding to the data volume which can be written in batch. At this time, the parameter of the data volume written in batch in the target writing thread is the data volume which can be written in batch by the idle node.

And 303, writing the update data to the idle node in batch through the target write thread.

In the embodiment of the application, when the updated data is written into the idle node through the target write-in thread, the effect of writing the updated data in batches can be achieved, and meanwhile, the data volume written in batches each time is related to the state information of the idle node, so that the efficiency of data updating can be improved to the maximum extent, and the overload of the node is avoided.

It should be noted that, as an alternative of the present application, in the process of writing the update data, the state information of the node may still be continuously obtained, and the batch writable data volume of each node is continuously updated.

The application provides an implementation mode for writing update data into idle nodes in batches through target write-in threads, the corresponding data volume which can be written in batches is determined through state information of the idle nodes, and the corresponding target write-in threads are established by taking the data volume as a write-in parameter, so that the update data can be written into the idle nodes in batches through the target write-in threads, the data updating efficiency is improved, meanwhile, the data volume written in batches each time is related to the state information of the idle nodes, and the overload of the nodes can be effectively avoided.

As shown in fig. 4, fig. 4 is a schematic flowchart of a third embodiment of a data updating method provided in the embodiment of the present application.

In the embodiment of the present application, an implementation manner for determining a batch writable data size of an idle node according to state information of the idle node is further provided, and specifically includes steps 401 to 405:

for each state information, a state threshold corresponding to the state information is obtained 401.

In the embodiment of the present application, when there are multiple pieces of state information, at this time, a corresponding state threshold is obtained for each piece of state information. Specifically, as a possible embodiment of the present application, for example, the state information includes the number of connection requests, the CPU occupancy rate, the memory occupancy rate, and the number of active threads at the same time, where the state threshold corresponding to the number of connection requests is 60, the state threshold corresponding to the memory occupancy rate is 4%, the state threshold corresponding to the CPU occupancy rate is 4%, and the state threshold corresponding to the number of active threads is 4.

Certainly, the setting of the state threshold is only a feasible scheme provided in the embodiment of the present application, and the specific setting of the state threshold may be set by a user according to actual requirements.

402, the ratio of the state information to the state threshold is set to the state level corresponding to the state information.

In the embodiment of the present application, in combination with the feasible embodiments provided above, that is, every time there are 60 request connections, the status level corresponding to the number of request connections is raised by 1, and similarly, every time there is a CPU occupancy of 4%, the status level corresponding to the CPU occupancy is raised by 1, every time there is a memory occupancy of 4%, the status level corresponding to the memory occupancy is raised by 1, and every time there are 4 active threads, the status level corresponding to the number of active threads is raised by 1. Specifically, the mapping relationship between each state information and the state level may be as follows:

1) Number of requested connections

If 0-59 nodes have connection requests, the state grade is 1;

if the nodes have 60 to 119 request connections, the state level is 2;

if 120-179 nodes have connection requests, the state level is 3;

if 180-239 request connections exist in the node, the state grade is 4;

if 240-299 request connections exist in the node, the state level is 5.

2) Memory usage rate

If the memory utilization rate of the node is 0-3%, the state grade is 1;

if the memory utilization rate of the node is 4-7%, the state grade is 2;

if the memory utilization rate of the node is 8-11%, the state grade is 3;

if the memory utilization rate of the node is 12-15%, the state grade is 4;

if the memory usage of the node is 16-19%, the status level is 5.

3) CPU utilization

If the CPU utilization rate of the node is 0-3%, the state grade is 1;

if the CPU utilization rate of the node is 4-7%, the state grade is 2;

if the CPU utilization rate of the node is 8-11%, the state grade is 3;

if the CPU utilization rate of the node is 12-15%, the state grade is 4;

if the CPU utilization of the node is 16 to 19%, the status level is 5.

4) Number of active threads

If the active thread number of the node is 0-3, the state grade is 1;

if the number of active threads of the node is 4-7, the state grade is 2;

if the number of active threads of the node is 8-11, the state grade is 3;

if the number of active threads of the node is 12-15, the state grade is 4;

if the number of active threads of a node is 16 to 19, the status level is 5.

Of course, the mapping relationship between the provided state information and the state level is only a feasible way, and the specific mapping relationship can be set by the user according to the actual situation.

As can be seen from the foregoing description, the highest status rank corresponding to each status information is set to 5. In fact, since the idle node is also determined by the state information, as an alternative embodiment of the present application, the maximum value corresponding to the maximum state level may also be understood as an idle threshold for determining the idle node. Specifically, in the step of determining a plurality of idle nodes from the nodes of the target cluster in step 203 according to the state information of each node, an idle threshold of each state information may be set, and a node whose state information is smaller than the corresponding maximum idle threshold is used as an idle node, that is, a node whose number of requested connections is smaller than 300, whose memory usage rate is smaller than 20%, whose CPU usage rate is smaller than 20%, and whose number of active threads is smaller than 20 is used as an idle node. Of course, the idle threshold specifically used for determining the idle node is not necessarily required to be equal to the maximum value corresponding to the maximum state level.

And 403, calculating to obtain an idle level corresponding to the idle node according to the state level corresponding to each state information.

In the embodiment of the application, after the state grade corresponding to each state information is obtained, the idle grade corresponding to the idle node can be further obtained by calculation according to the state grade corresponding to each state information. For example, the simplest one may determine the maximum state level among the state levels corresponding to the respective state information as the idle level corresponding to the idle node.

As another feasible scheme, the state levels corresponding to the state information may be weighted and summed according to a preset weight, so as to obtain idle levels corresponding to idle nodes. Specifically, for example, the weight corresponding to the number of requested connections is 0.35, the weight corresponding to the memory occupancy rate is 0.25, the weight corresponding to the cpu occupancy rate is 0.2, and the weight corresponding to the number of active threads is 0.2, and the state grades corresponding to the state information are weighted and summed according to the weights, so that the idle grade corresponding to the idle node can be obtained. Of course, the setting of the above-mentioned weight is only a feasible solution, and the specific weight can be set by the user according to the actual situation.

In the embodiment of the present application, taking the specific scheme provided above as an example, at this time, the idle level corresponding to the idle node is also generally divided into 5 levels as the state level.

404, querying a preset database to obtain a target data volume corresponding to the idle level.

In the embodiment of the present application, it can be known by combining the foregoing description that the idle level corresponding to the idle node is also generally divided into 5 levels as the state level, and it is easy to understand that the higher the idle level is, that is, the number of requested connections, the memory usage rate, the CPU usage rate, and the number of active threads are often higher, and at this time, the load of the node itself is often higher, so when the data volume written in batch is set, the data volume should be smaller. Specifically, as a feasible scheme, the preset database has the following correspondence between the idle level and the target data volume:

when the idle level is 1, the corresponding target data size is 5000;

when the idle level is 2, the corresponding target data volume is 4000;

when the idle level is 3, the corresponding target data size is 3000;

when the idle level is 4, the corresponding target data amount is 2000;

when the idle level is 5, the corresponding target data amount is 1000.

Similarly, the correspondence between the idle level and the target data amount is only a feasible scheme, and the specific correspondence may also be determined by the user according to the actual requirement, but regardless of the specific correspondence, the target data amount needs to be negatively correlated with the idle level, that is, the higher the idle level of the node is, the higher the load of the node itself is, the lower the target data amount is.

The target data volume is set to the bulk writable data volume of the free node 405.

In the embodiment of the present application, in a general case, the target data volume obtained by querying the preset database is the data volume that can be written in batch for the idle node.

The embodiment of the application provides an implementation method for determining the batch-writable data volume of idle nodes according to the state information of the idle nodes, the idle levels of the idle nodes are obtained through calculation according to the state information, then the target data volume corresponding to the idle levels is obtained through query, and compared with the method that the incidence relation between the target data volume and each state information is directly stored in a database, the method and the device for determining the batch-writable data volume of the idle nodes have lower storage requirements on the database.

As shown in fig. 5, fig. 5 is a schematic flowchart of a fourth embodiment of a data updating method provided in the embodiment of the present application.

In the embodiment of the present application, an implementation manner of writing specified update data into an idle node through a write thread is further provided, and specifically includes steps 501 to 504:

501, obtaining the storage space utilization rate of a free node.

In the embodiment of the present application, it can be known from the foregoing description that, taking an elastic search cluster as an example, the storage space utilization rate of a node may also be obtained by a corresponding node, that is, the storage space utilization rate may also be understood as one of the obtained state information of the node. The present invention is not described herein in detail.

And 502, determining the data write share of the idle node according to the storage space utilization rate of the idle node.

In the embodiment of the present application, the data write share refers to a ratio of a data amount that needs to be written by an idle node to a data amount of all update data.

The higher the storage space usage rate of a node is, the smaller the data amount that the node can store is indicated, and the lower the storage space usage rate is, the larger the data amount that the node can store is indicated, so that a data write share of an idle node can be further determined according to the storage space usage rate of the idle node, where the data write share is inversely proportional to the storage space usage rate of the idle node, that is, the lower the storage space usage rate of a certain node is, the more data can be written into the node, and thus, the larger the data write share is.

As another alternative of the present application, the data write share is not only related to the storage space utilization rate of the idle node, but also positively related to the data volume of batch writing of the idle node, that is, the larger the data volume of batch writing of the idle node is, the larger the data write share of the idle node is. Specifically, the data write share of the idle node is determined by the storage space usage rate of the idle node and the data volume written in batches by the idle node, and usually, the association relationship may be stored in the database in advance. The specific implementation manner for determining the data write share jointly according to the storage space utilization rate of the idle node and the data volume written in batches by the idle node is not described in detail in the present application.

503, dividing the update data according to the data write share of the idle node to obtain the node update data corresponding to the idle node.

In the embodiment of the application, after the data write share of the idle node is obtained, the update data is divided according to the data write partition of each idle node, and then the node update data corresponding to each idle node can be obtained.

Node update data is written to the idle node by the write thread 504.

In the embodiment of the application, after the node update data corresponding to the idle node is obtained, the corresponding node update data can be written into the idle node through the write-in thread.

As an alternative to this application, the embodiments provided in this application may further combine with the embodiments provided in fig. 3 and fig. 4, so as to obtain an implementation scheme for writing node update data to an idle node in bulk by a write thread, where bulk writing is implemented by the embodiment provided in fig. 3, and writing corresponding node update data is implemented by the embodiments of this application. The detailed description of the specific implementation scheme is omitted here.

In the embodiment of the application, a technical scheme that a data write-in share of each idle node and corresponding node update data are determined according to the storage space utilization rate of the idle node, so that the corresponding node update data are written into the idle node is provided. The cluster data to be written in is divided according to the writable data writing share of each idle node in advance, so that the phenomenon that a certain idle node writes in excessive data to cause that the storage space is completely occupied is avoided, and the data updating effect is improved.

As shown in fig. 6, fig. 6 is a schematic flowchart of a fifth embodiment of a data updating method provided in the embodiment of the present application.

In the embodiment of the present application, an implementation manner for determining a plurality of idle nodes from each node according to state information of each node is provided, which specifically includes steps 601 to 603:

601, an idle threshold corresponding to the state information is obtained.

In the embodiment of the present application, as can be known from the related description of the foregoing step 402, by presetting the idle threshold corresponding to each piece of state information, it may be used to determine whether each node is in an idle state.

Specifically, the method comprises the following steps. The state information includes the number of request connections, CPU occupancy, memory occupancy, and the number of active threads. As a feasible implementation manner of the present application, at this time, the idle threshold corresponding to the number of requested connections is 300, the idle threshold corresponding to the memory occupancy rate is 20%, the idle threshold corresponding to the CPU occupancy rate is 20%, and the number of active threads corresponding to the number of active threads is 20.

For each node in the target cluster, the state information of the node is compared with an idle threshold 602.

In the embodiment of the application, for each node, the state information of the node is compared with the corresponding idle threshold. For example, the current request connection number, the current CPU occupancy rate, the current memory occupancy rate, and the current active thread number of the node are respectively compared with a preset idle threshold, to determine the size relationship between the state information of the node and the idle threshold, that is, the current request connection number of the node is compared with 300, the current CPU occupancy rate of the node is compared with 20%, the current memory occupancy rate of the node is compared with 20%, and the current active thread number of the node is compared with 20.

If the state information of the node includes a plurality of quantities, it is necessary to compare each piece of state information with the corresponding idle threshold.

603, if the state information of the node is less than the idle threshold, setting the node as an idle node.

In this embodiment of the present application, specifically, when the current requested connection number of a certain node is less than 300, the current CPU occupancy rate is less than 20%, the current memory occupancy rate is less than 20%, and the current active thread number is less than 20, the node may be set as an idle node. If the state information includes a plurality of quantities, then each quantity is required to be less than the corresponding idle threshold.

The embodiment of the application provides an implementation mode for determining a plurality of idle nodes from nodes of a target cluster according to state information of each node, and the nodes are set as the idle nodes by setting idle thresholds corresponding to the state information and comparing the state information of the nodes with the idle thresholds when the state information of the nodes is smaller than the idle thresholds.

As shown in fig. 7, fig. 7 is a schematic flowchart of a sixth embodiment of a data updating method provided in the embodiment of the present application.

In view of that each time the state information of the nodes in the cluster needs to occupy a certain cluster resource, in this embodiment of the present application, a scheme for starting data update is provided, specifically, before obtaining the state information of each node in the target cluster, a data amount of update data is compared with a preset data amount threshold, and when a specific requirement is met, a subsequent step of updating data is executed to save the cluster resource, specifically including steps 701 to 703:

701, comparing the data volume of the update data with a preset data volume threshold.

In the embodiment of the application, after the data volume of the update data is obtained, the data volume of the update data is compared with a preset data volume threshold value, so as to judge the size relationship between the data volume of the update data and the data volume threshold value. The preset data amount threshold may be understood as a minimum data amount requirement for updating data.

And 702, if the data volume of the updated data is larger than the data volume threshold value, executing the step of acquiring the state information of each node in the target cluster.

In the embodiment of the application, when the data volume of the updated data is greater than the preset data volume threshold, the data updating device may perform the step of obtaining the state information of each node in the target cluster to further perform the subsequent step of updating the data. And when the data volume of the updated data is less than or equal to the preset data volume threshold, the data updating device does not acquire the state information of each node in the target cluster, and the subsequent data updating step is not executed.

According to the scheme provided by the embodiment of the application, after the update data to be written into the target cluster is obtained, the data volume of the update data is compared with the preset data volume threshold, and the state information of each node in the target cluster is further obtained only when the data volume of the update data is larger than the preset data volume threshold. By the method, the data updating process can be executed only after the updating data reaches a certain data volume, so that the situation that the state information of each node is repeatedly acquired for many times and cluster resources are occupied is avoided.

As shown in fig. 8, fig. 8 is a schematic flowchart of a seventh embodiment of a data updating method provided in the embodiment of the present application.

Similarly, considering that the data updating apparatus needs to call resources when acquiring the data to be updated, and in order to prevent the data updating apparatus from repeatedly acquiring the update data of the target cluster, in this embodiment of the present application, another scheme for starting data updating is provided, specifically, a time for acquiring the update data is determined by using the time information, including steps 801 to 802:

the obtained time information is compared 801 with a preset update period.

In the embodiment of the present application, the time information obtained by the data updating apparatus is the current time information, and may be specifically obtained by calling an interface, which is not described herein again.

In the embodiment of the application, whether the current time is within the updating time period is judged by comparing the time information with the preset updating time period through the preset updating time period.

Specifically, the update period is determined in advance according to the peak period of the cluster, that is, a period other than the peak period of the cluster is set as the update period, in other words, the update period can also be understood as the low peak period of the cluster.

And 802, if the time information is in the updating time period, acquiring the updating data to be written into the target cluster.

In the embodiment of the application, if the time information is in the updating period, that is, the current time does not belong to the peak period of the cluster, at this time, each node in the cluster is in an idle state with higher possibility, and at this time, the subsequent data updating method is executed, so that the data can be updated better.

In the embodiment of the application, by comparing the acquired time information with the preset updating time period, data updating can be performed in the idle time period of the system, and meanwhile, the situation that the updating data to be written into the target cluster is repeatedly acquired is avoided, and the effect of subsequent data updating is effectively improved.

In order to better implement the data updating method provided in the embodiment of the present application, based on the data updating method, a data updating device is further provided in the embodiment of the present application, as shown in fig. 9, and fig. 9 is a schematic structural diagram of the data updating device provided in the embodiment of the present application.

In an embodiment of the present application, a data updating apparatus includes:

a first obtaining module 901, configured to obtain update data to be written into a target cluster;

a second obtaining module 902, configured to obtain state information of each node in the target cluster;

a node determining module 903, configured to determine a plurality of idle nodes from each node according to the state information of each node;

and a writing module 904, configured to create a write thread for each idle node, and write update data into the idle node through the write thread to obtain state information of each node in the target cluster to be updated.

In some embodiments of the present application, the writing module includes:

the batch write data volume determining secondary module is used for determining the batch write data volume of the idle node according to the state information of the idle node;

the target writing thread creating submodule is used for creating a target writing thread corresponding to the data volume by taking the data volume as a writing parameter of the writing thread;

and the first write-in submodule is used for writing the update data into the idle nodes in batch through the target write-in thread.

In some embodiments of the application, the batch write data amount determining submodule includes:

a state threshold acquisition unit configured to acquire, for each piece of state information, a state threshold corresponding to the state information;

the state grade setting unit is used for setting the ratio of the state information to the state threshold value as the state grade corresponding to the state information;

the idle level determining unit is used for calculating and obtaining an idle level corresponding to the idle node according to the state level corresponding to each state information;

the data volume query unit is used for querying a preset database and acquiring a target data volume corresponding to the idle level;

and the data volume setting unit is used for setting the target data volume as the data volume which can be written in batches of the idle nodes.

In some embodiments of the present application, the writing module includes:

the space utilization rate obtaining secondary module is used for obtaining the storage space utilization rate of the idle node;

the write-in share determining secondary module is used for determining the data write-in share of the idle node according to the storage space utilization rate of the idle node;

the data dividing sub-module is used for dividing the update data according to the data write-in share of the idle node to obtain node update data corresponding to the idle node;

and the second write-in secondary module is used for writing the node update data into the idle node through the write-in thread.

In some embodiments of the present application, the data updating apparatus includes:

the first comparison module is used for comparing the data volume of the updated data with a preset data volume threshold;

the second obtaining module is configured to obtain the state information of each node in the target cluster if the data amount of the update data is greater than the data amount threshold.

In some embodiments of the present application, the data updating apparatus includes:

the second comparison module is used for comparing the obtained time information with a preset updating time period;

the first obtaining module is configured to obtain update data to be written into the target cluster if the time information is located in the update period.

An embodiment of the present invention further provides a data updating apparatus, as shown in fig. 10, fig. 10 is a schematic structural diagram of an embodiment of the data updating apparatus provided in the embodiment of the present application.

The data updating device comprises a memory, a processor and a data updating program which is stored in the memory and can run on the processor, and the steps in the data updating method in any embodiment of the application are realized when the processor executes the data updating program.

Specifically, the method comprises the following steps: the data update apparatus may include components such as a processor 1001 of one or more processing cores, a memory 1002 of one or more storage media, a power supply 1003, and an input unit 1004. Those skilled in the art will appreciate that the data update apparatus configuration shown in FIG. 10 does not constitute a limitation of the data update apparatus, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 1001 is a control center of the data update apparatus, connects various parts of the entire data update apparatus using various interfaces and lines, and performs various functions of the data update apparatus and processes data by running or executing software programs and/or modules stored in the memory 1002 and calling the data stored in the memory 1002, thereby performing overall monitoring of the data update apparatus. Optionally, processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 1001.

The memory 1002 may be used to store software programs and modules, and the processor 1001 executes various functional applications and data processing by operating the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the data update apparatus, and the like. Further, the memory 1002 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 access to the memory 1002.

The data update apparatus further includes a power source 1003 for supplying power to each component, and preferably, the power source 1003 may be logically connected to the processor 1001 through a power management system, so that functions of managing charging, discharging, power consumption management, and the like are implemented through the power management system. The power source 1003 may also include any component including one or more of a dc or ac power source, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

The data update apparatus may further include an input unit 1004, and the input unit 1004 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the data update apparatus may further include a display unit and the like, which will not be described in detail herein. Specifically, in this embodiment, the processor 1001 in the data updating apparatus loads the executable file corresponding to the process of one or more applications into the memory 1002 according to the following instructions, and the processor 1001 runs the applications stored in the memory 1002, thereby implementing the steps in any data updating method provided by the embodiment of the present invention.

To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like. The computer readable storage medium stores a data updating program, and the data updating program is used for implementing the steps of any data updating method provided by the embodiment of the invention when being executed.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed descriptions of other embodiments, and are not described herein again.

In specific implementation, each unit or structure may be implemented as an independent entity, or may be combined arbitrarily to be implemented as the same entity or several entities, and specific implementation of each unit or structure may refer to the foregoing method embodiment, which is not described herein again.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

The above detailed description of a data updating method provided in the embodiments of the present application has applied specific examples to explain the principle and the implementation of the present invention, and the above description of the embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (10)

1. A method for updating data, comprising:

acquiring update data to be written into a target cluster;

acquiring state information of each node in the target cluster;

determining a plurality of idle nodes from each node according to the state information of each node;

and creating a write thread for each idle node, and writing the update data into the idle node through the write thread.

2. The method of claim 1, wherein the status information comprises at least one of a number of requested connections, a CPU occupancy, a memory occupancy, and a number of active threads;

the creating a write thread and writing the update data to the idle node through the write thread includes:

determining the data volume which can be written in batch of the idle nodes according to the state information of the idle nodes;

taking the data volume as a writing parameter of a writing thread, and creating a target writing thread corresponding to the data volume;

and writing the update data to the idle node in batch through the target write thread.

3. The method of claim 2, wherein the determining the amount of bulk writable data of the idle node according to the status information of the idle node comprises:

acquiring a state threshold corresponding to each state information;

setting the ratio of the state information to the state threshold value as a state grade corresponding to the state information;

calculating to obtain an idle grade corresponding to the idle node according to the state grade corresponding to each state information;

querying a preset database to obtain a target data volume corresponding to the idle level;

and setting the target data volume as the batch writable data volume of the idle node.

4. The method of claim 1, wherein the writing the update data to the idle node by the write thread comprises:

acquiring the storage space utilization rate of the idle node;

determining the data write-in share of the idle node according to the storage space utilization rate of the idle node;

dividing the update data according to the data write-in share of the idle node to obtain node update data corresponding to the idle node;

and writing the node updating data into the idle node through the write thread.

5. The method of claim 4, wherein determining a number of idle nodes from each of the nodes based on the state information of each of the nodes comprises:

acquiring an idle threshold corresponding to the state information;

comparing the state information of the nodes with the idle threshold value aiming at each node in the target cluster;

and if the state information of the node is smaller than the idle threshold value, setting the node as an idle node.

6. The method according to any one of claims 1 to 5, wherein before the obtaining the state information of each node in the target cluster, the method further comprises:

comparing the data volume of the updated data with a preset data volume threshold;

and if the data volume of the updated data is larger than the data volume threshold, executing the step of acquiring the state information of each node in the target cluster.

7. The method according to any of claims 1 to 5, wherein before obtaining the update data to be written to the target cluster, the method further comprises:

comparing the obtained time information with a preset updating time period;

and if the time information is located in the updating time period, executing the step of acquiring the updating data to be written into the target cluster.

8. A data update apparatus, comprising:

the first acquisition module is used for acquiring the update data to be written into the target cluster;

the second acquisition module is used for acquiring the state information of each node in the target cluster;

a node determining module, configured to determine a plurality of idle nodes from each of the nodes according to the state information of each of the nodes;

and the writing module is used for creating a writing thread aiming at each idle node, and writing the updating data into the idle node through the writing thread to obtain the state information of each node in the target cluster to be updated.

9. A data update apparatus, characterized in that the data update apparatus comprises a processor, a memory, and a data update program stored in the memory and executable on the processor, the processor executing the data update program to implement the steps in the data update method of any one of claims 1 to 7.

10. A computer-readable storage medium having a data update program stored thereon, the data update program being executed by a processor to implement the steps in the data update method of any one of claims 1 to 7.

Images