CN113722292B - Disaster response processing method, device, equipment and storage medium of distributed data system - Google Patents
Disaster response processing method, device, equipment and storage medium of distributed data system Download PDFInfo
- Publication number
- CN113722292B CN113722292B CN202111007167.7A CN202111007167A CN113722292B CN 113722292 B CN113722292 B CN 113722292B CN 202111007167 A CN202111007167 A CN 202111007167A CN 113722292 B CN113722292 B CN 113722292B
- Authority
- CN
- China
- Prior art keywords
- distributed data
- data system
- resource
- utilization rate
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004044 response Effects 0.000 title claims abstract description 108
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 238000012545 processing Methods 0.000 claims abstract description 62
- 238000000034 method Methods 0.000 claims abstract description 56
- 238000000137 annealing Methods 0.000 claims abstract description 15
- 230000006870 function Effects 0.000 claims description 101
- 230000008569 process Effects 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 23
- 238000011084 recovery Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 abstract description 8
- 238000013473 artificial intelligence Methods 0.000 abstract description 7
- 238000005516 engineering process Methods 0.000 description 13
- 230000010485 coping Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000013523 data management Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000002922 simulated annealing Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003094 perturbing effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application is applicable to the technical field of artificial intelligence and system disaster response processing, and provides a disaster response processing method, device, equipment and storage medium of a distributed data system, wherein the method comprises the following steps: acquiring the current resource utilization rate of the distributed data system, and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold; if the resource utilization rate reaches the disaster response starting threshold, solving the consumption of the computing resource which needs to be added in the disaster response of the system by adopting a preset disaster response model; and carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resources which are required to be added, so that the resource utilization rate of the distributed data system is reduced to a reference value representing the safe operation of the system. Based on the method, when the distributed data system has a sudden data disaster event, the annealing algorithm can be utilized to perform data disaster 'pouring-out', so that the resource utilization rate of clusters in the distributed data system is reduced to a safe value, and the normal operation of the system is ensured.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence and disaster response processing technologies for systems, and in particular, to a disaster response processing method, apparatus, device, and storage medium for a distributed data system.
Background
In the data age of big data with the rapid growth of data, big data mining and analysis technology is widely applied in various industry fields. However, with the increasing data volume, it is difficult for the conventional data management system to ensure the security of data when there is a sudden data disaster event, and the capability of coping with the data disaster and the sudden data loss is lacking.
Disclosure of Invention
In view of this, the embodiments of the present application provide a disaster response processing method, apparatus, electronic device, and storage medium for a distributed data system, which can effectively improve the capability of the distributed data system to cope with data disaster and sudden data loss while implementing distributed storage of data by fusing Hadoop cluster technology and annealing algorithm, thereby solving the problems of reading efficiency and data security existing in conventional medical large data storage.
A first aspect of an embodiment of the present application provides a disaster response processing method for a distributed data system, including:
acquiring the current resource utilization rate of a distributed data system, and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold;
if the resource utilization rate reaches a disaster response starting threshold, solving the consumption of computing resources required to be added in the disaster response of the system by adopting a preset disaster response model;
and carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resource to be added so as to reduce the resource utilization rate of the distributed data system to a reference value representing the safe operation of the system.
With reference to the first aspect, in a first possible implementation manner of the first aspect, the step of solving, by using a preset disaster response model, a calculation resource usage amount that needs to be added in response to a system disaster, if the resource usage rate reaches a disaster response start threshold includes:
inputting a preset initialized computing resource additional consumption into the preset disaster response model for iterative processing to obtain an iterative computing resource additional consumption;
according to the iterative calculation resource adding amount, simulating and calculating the resource utilization rate of the distributed data system after adding the calculation resource;
judging whether the resource utilization rate of the distributed data system after adding the computing resource is lower than a preset resource utilization rate threshold value, if yes, configuring the iterative computing resource adding amount as the computing resource amount required to be added by the system disaster, otherwise, carrying out iterative processing again until the resource utilization rate of the distributed data system after adding the computing resource is lower than the preset resource utilization rate threshold value.
With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the step of inputting a preset initialized additional usage of computing resources into the preset disaster response model to perform iterative processing, and obtaining the iterative additional usage of computing resources includes:
the preset initialized calculation resource adding amount is used as an initial solution of a target loss function in the preset disaster response model to calculate the target loss function, and a first function value is obtained;
carrying out disturbance processing on the initial solution according to preset disturbance conditions to obtain a plurality of disturbance solutions corresponding to the initial solution;
randomly selecting one disturbance solution from the disturbance solutions as a new solution of a target loss function in the preset disaster response model to calculate the target loss function, and obtaining a second function value;
and comparing the first function value with the second function value, if the second function value is smaller than or equal to the first function value, receiving the new solution, and performing next round of iterative processing on the target loss function by taking the new solution as an initial solution until the iterative times reach a preset time threshold value, so as to obtain the calculated resource additional consumption after iteration.
With reference to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, in the step of performing a perturbation process on the initial solution according to a preset perturbation condition to obtain a plurality of perturbation solutions corresponding to the initial solution, the preset perturbation condition is configured to:
ω 2 -ω’>0 and ω' 2 ＝2-ω
Where ω is denoted as the initial solution and ω' is denoted as the disturbance solution.
With reference to the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, after the step of comparing the first function value and the second function value, if the second function value is greater than the first function value, the method further includes:
calculating a probability value for accepting the new solution according to a Metropolis criterion according to the first function value and the second function value;
and comparing the calculated probability value with a preset probability threshold, if the probability value is larger than the preset probability threshold, accepting the new solution, and carrying out next round of iterative processing on the target loss function by taking the new solution as an initial solution, otherwise, not accepting the new solution.
With reference to the first possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, if the resource usage rate of the distributed data system after adding the computing resource is not lower than a preset resource usage rate threshold, performing iterative processing again until the resource usage rate of the distributed data system after adding the computing resource is lower than the preset resource usage rate threshold, the method includes:
and reducing the resource utilization rate of the distributed data system according to a preset annealing rate and resetting the iteration times so as to carry out iteration processing again based on the annealed resource utilization rate of the distributed data system.
With reference to the first aspect or any one of possible implementation manners of the first aspect, in a sixth possible implementation manner of the first aspect, the performing, by using the computing resource to be added, resource scheduling processing on the distributed data system, so that a resource usage rate of the distributed data system falls to a reference value indicating safe operation of the system, where a manner of performing resource scheduling processing includes internal scheduling processing and external access processing, where: the internal scheduling process is resource scheduling between slave nodes in the distributed data system, and the external access process is to access computing resources from outside of the distributed data system.
A second aspect of an embodiment of the present application provides a disaster response processing device of a distributed data system, where the disaster response processing device of the distributed data system includes:
the judging module is used for acquiring the current resource utilization rate of the distributed data system and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold value or not;
the calculation module is used for solving the consumption of the calculation resources which are needed to be added in the system disaster response by adopting a preset disaster response model if the resource utilization rate reaches a disaster response starting threshold;
and the scheduling module is used for carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resources to be added so as to reduce the resource utilization rate of the distributed data system to a reference value representing the safe operation of the system.
A third aspect of the embodiments of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the electronic device, where the processor implements the steps of the disaster response processing method of the distributed data system provided in the first aspect when the processor executes the computer program.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the steps of the disaster response processing method of the distributed data system provided in the first aspect.
The disaster response processing method, device, electronic equipment and storage medium of the distributed data system have the following beneficial effects:
the method comprises the steps of obtaining the current resource utilization rate of the distributed data system, and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold value or not; if the resource utilization rate reaches the disaster response starting threshold, solving the consumption of the computing resource which needs to be added in the disaster response of the system by adopting a preset disaster response model; and carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resources which are required to be added, so that the resource utilization rate of the distributed data system is reduced to a reference value representing the safe operation of the system. Based on the method, an annealing algorithm is fused in a distributed data system constructed by adopting a Hadoop cluster technology through a preset disaster response model, so that the capacity of the distributed data system for coping with difficult data disaster and sudden data loss is effectively improved while the distributed data system is subjected to distributed storage, and therefore, when the distributed data system has sudden data disaster events, the annealing algorithm is utilized to perform 'pouring and extinguishing' of the data disaster, so that the resource utilization rate of clusters in the distributed data system is reduced to a safe value, and the normal operation of the distributed data system is ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required for the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of an implementation of a disaster response method for a distributed data system according to an embodiment of the present application;
FIG. 2 is a flow chart of a process when a distributed data system stores data in a disaster response processing method of the distributed data system according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of a method for solving the computing resource consumption needed to be added when a system disaster is solved in the disaster response processing method of the distributed data system according to the embodiment of the present application;
fig. 4 is a schematic flow chart of a method for obtaining an additional usage amount of iterative computing resources through iteration in the disaster response processing method of the distributed data system according to the embodiment of the present application;
FIG. 5 is a flowchart illustrating another method for obtaining an additional usage amount of iterative computing resources by iteration in the disaster response processing method of the distributed data system according to the embodiment of the present application;
FIG. 6 is a basic block diagram of a disaster response device of a distributed data system according to an embodiment of the present application;
fig. 7 is a basic structural block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a disaster response method for a distributed data system according to an embodiment of the present application. The details are as follows:
s11: and acquiring the current resource utilization rate of the distributed data system, and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold.
In this embodiment, the distributed data system is a system for data storage and management built by using Hadoop (distributed system infrastructure). At least one master node and a plurality of slave nodes are built in the distributed data system, wherein the master node is used for executing work for coping with data disasters, and the slave nodes are used for storing data of different categories. For example, referring to fig. 2 together, fig. 2 is a flowchart illustrating a process when a distributed data system stores data in a disaster response processing method of the distributed data system according to an embodiment of the present application. As shown in fig. 2, the distributed data system provides an online access portal and an offline access portal, wherein the online portal is directly connected with the database, supports reading data directly from the database, the offline portal supports reading a local csv file, reads data from the local csv file, and gathers the data read by the two access portals after reading the data from the two different access portals. Then, according to the read data, data slicing processing is carried out according to the set data types, and the data is divided into a plurality of data blocks. Further, the plurality of data blocks are stored in different slave nodes according to data types.
In this embodiment, a higher resource usage rate of a node indicates that the node has a higher probability of being down, which results in a data disaster in which the data of the node cannot be used. In this embodiment, a disaster response starting threshold for triggering the system to execute disaster response processing may be set in the distributed data system in advance, where the disaster response starting threshold is a resource usage upper limit reference value that characterizes that the system can normally operate. And then monitoring the resource utilization rate of each slave node at the master node of the distributed data system, and summarizing and integrating the resource utilization rate of each slave node to obtain the current resource utilization rate of the distributed data system. And comparing the obtained current resource utilization rate of the distributed data system with a disaster response starting threshold preset in the system, judging whether the obtained current resource utilization rate of the distributed data system is larger than or equal to the disaster response starting threshold preset in the system, and if so, judging whether the current resource utilization rate of the distributed data system reaches the disaster response starting threshold.
S12: and if the resource utilization rate reaches the disaster response starting threshold, solving the consumption of the computing resource which needs to be added in the disaster response of the system by adopting a preset disaster response model.
In this embodiment, a preset disaster response model is constructed by using a simulated annealing algorithm, and in the disaster response model, a target loss function is designed, and the target loss function adopts a cross entropy loss function, which is specifically as follows:
L＝-[ylog y^+(1-y)log(1-y^)]
wherein y is represented as the true output of the disaster response model, and y is represented as the predicted output of the disaster response model.
In this embodiment, the input of the disaster response model is the initially set computing resource addition amount, and the output of the disaster response model is the resource utilization rate of the system after adding the computing resource. In the disaster response model, based on the input computing resource additional consumption, iterating the target loss function, and solving the computing resource consumption required to be added in the disaster response of the system by minimizing the target loss function. In this embodiment, when the resource usage rate output by the disaster response model reaches the reference value of the safe operation of the system, it means that the objective loss function is minimized.
S13: and carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resource to be added so as to reduce the resource utilization rate of the distributed data system to a reference value representing the safe operation of the system.
In this embodiment, the resource scheduling processing includes two modes, i.e., internal scheduling processing and external access processing. Wherein: the internal scheduling process is the resource scheduling between the slave nodes in the distributed data system, and the external access process is the external access computing resource of the distributed data system. That is, the resource scheduling process may be performed from an existing computing resource within the distributed data system, or may be performed by adding a new computing resource from the outside. In this embodiment, when the internal scheduling processing manner is used, the computing resources in the slave nodes with sufficient computing resources are scheduled to the slave nodes with insufficient computing resources by specifically calculating the computing resource usage condition of each slave node in the distributed data system, so as to implement internal scheduling. It can be understood that when the internal scheduling process is performed, it is necessary to ensure that the resource utilization rate of the slave node itself with sufficient computing resources can be stabilized below a reference value representing the safe operation of the system, and then schedule the remaining computing resources to other slave nodes.
As can be seen from the above, the disaster response processing method for a distributed data system provided by the present embodiment obtains the current resource usage rate of the distributed data system, and determines whether the current resource usage rate of the distributed data system reaches the disaster response starting threshold; if the resource utilization rate reaches the disaster response starting threshold, solving the consumption of the computing resource which needs to be added in the disaster response of the system by adopting a preset disaster response model; and carrying out resource scheduling processing on the distributed data system according to the consumption of the computing resources which are required to be added, so that the resource utilization rate of the distributed data system is reduced to a reference value representing the safe operation of the system. By means of a preset disaster response model, an annealing algorithm is fused in a distributed data system constructed by adopting a Hadoop cluster technology, the capacity of the distributed data system for coping with difficult data disaster and sudden data loss is effectively improved while the distributed data system is subjected to distributed storage, so that when the distributed data system has sudden data disaster events, the annealing algorithm is utilized to perform 'pouring and extinguishing' of the data disaster, the resource utilization rate of clusters in the distributed data system is reduced to a safe value, and normal operation of the distributed data system is guaranteed.
In some embodiments of the present application, referring to fig. 3, fig. 3 is a flow chart of a method for solving a computing resource consumption needed for system disaster recovery in the disaster recovery processing method of the distributed data system provided in the embodiments of the present application. The details are as follows:
s31: inputting a preset initialized computing resource additional consumption into the preset disaster response model for iterative processing to obtain an iterative computing resource additional consumption;
s32: according to the iterative calculation resource adding amount, simulating and calculating the resource utilization rate of the distributed data system after adding the calculation resource;
s33: judging whether the resource utilization rate of the distributed data system after adding the computing resource is lower than a preset resource utilization rate threshold value, if yes, configuring the iterative computing resource adding amount as the computing resource amount required to be added by the system disaster, otherwise, carrying out iterative processing again until the resource utilization rate of the distributed data system after adding the computing resource is lower than the preset resource utilization rate threshold value.
In this embodiment, the preset initialized additional usage of computing resources is obtained through random setting. In this embodiment, an initialized additional computing resource amount may be input at random, then a disturbance solution corresponding to the initialized additional computing resource amount may be generated by randomly perturbing the initialized additional computing resource amount, the objective loss function may be calculated by using the initialized additional computing resource amount and the disturbance solution corresponding to the initialized additional computing resource amount, so as to obtain two function values respectively corresponding to the two function values, and further by comparing the two function values, if the function value obtained when the objective loss function is calculated by using the disturbance solution corresponding to the initialized additional computing resource amount is smaller than or equal to the function value obtained when the objective loss function is calculated by using the initialized additional computing resource amount, the disturbance solution may be accepted as the additional computing resource amount after the iteration. In this embodiment, an iteration number threshold may be set in the disaster response model, so as to obtain the iterative computing resource additional usage through multiple iterations. It can be understood that when multiple iterations are performed, after the disturbance solution in the objective loss function iteration process is accepted, when the next iteration is performed, the disturbance solution is used as an initial solution to perform random disturbance again to generate a disturbance solution of the next iteration. After the iterative computing resource adding consumption is obtained, simulating and adding the iterative computing resource adding consumption into the distributed data system according to the iterative computing resource adding consumption, so as to simulate and calculate the resource utilization rate of the distributed data system after adding the computing resource. And further, judging whether the resource utilization rate of the distributed data system after adding the computing resource is lower than a preset resource utilization rate threshold value, so as to determine whether the iterative computing resource adding amount can be used as the computing resource amount which needs to be added for system disaster response. In this embodiment, the preset resource usage threshold is characterized as a reference value for safe operation of the system. If the resource utilization rate of the distributed data system after the additional computing resource is lower than the preset resource utilization rate threshold after the additional computing resource is simulated according to the iterative computing resource additional consumption, the iterative computing resource additional consumption is indicated to meet the disaster response requirement of the distributed data system, and at the moment, the iterative computing resource additional consumption can be configured as the computing resource consumption required to be added for the distributed data system disaster response.
In some embodiments of the present application, referring to fig. 4, fig. 4 is a flowchart of a method for obtaining an additional usage amount of iterative computing resources by iteration in the disaster response processing method of the distributed data system provided in the embodiments of the present application, which is described in detail below:
s41: calculating the target loss function by taking the preset initialized calculation resource adding amount as an initial solution of the target loss function in the preset disaster response model to obtain a first function value;
s42: carrying out disturbance processing on the initial solution according to preset disturbance conditions to obtain a plurality of disturbance solutions corresponding to the initial solution;
s43: randomly selecting one disturbance solution from the disturbance solutions as a new solution of a target loss function in the preset disaster response model to calculate the target loss function, and obtaining a second function value;
s44: and comparing the first function value with the second function value, if the second function value is smaller than or equal to the first function value, accepting the new solution, and carrying out iterative processing on the target loss function by taking the new solution as an initial solution until the iterative times reach a preset time threshold value, so as to obtain the iterative calculation resource adding amount.
In this embodiment, the initialized additional usage of computing resources is obtained by random setting. In this embodiment, the initialized additional usage of computing resources is set to be a usage of one computing resource unit, for example, 100m. After the initialized additional usage of the computing resource is obtained, in the disaster response model, the initialized additional usage of the computing resource can be used as an initial solution to calculate a target loss function in the disaster response model, so as to obtain a first function value. For the initialized additional computing resource consumption, a disturbance solution corresponding to the initialized additional computing resource consumption may be obtained by performing a disturbance process according to a preset disturbance condition, where it is understood that the disturbance solution is also characterized as the additional computing resource consumption. For example, assuming that the initialized computing resource additional usage is denoted as ω, the perturbation solution is denoted as ω', the perturbation condition may be set as: omega 2 -ω’>0 and ω' 2 =2- ω, a finite number of perturbation solutions can be generated based on the perturbation condition. In this embodiment, a disturbance solution may be randomly selected from the limited number of disturbance solutions as a new solution of the target loss function, and the target loss function may be calculated to obtain a second function value. After the first function value and the second function value are obtained, the first function value and the second function value are compared, if the second function value is smaller than or equal to the first function value, the disturbance solution is received, and the disturbance solution is used as an initial solution to perform next round of iterative processing on the target loss function. In this embodiment, an iteration number threshold is preset in the disaster response model, so that the number of iterations is multiple, until the number of iterations reaches the preset number threshold, to obtain the additional consumption of the iterative computing resource.
In some embodiments of the present application, referring to fig. 5, fig. 5 is a flowchart of another method for obtaining an additional usage of iterative computing resources by iteration in the disaster response processing method of the distributed data system according to the embodiment of the present application, which is described in detail below:
s51: calculating a probability value for accepting the new solution according to a Metropolis criterion according to the first function value and the second function value;
s52: and comparing the calculated probability value with a preset probability threshold, if the probability value is larger than the preset probability threshold, accepting the new solution, and carrying out next round of iterative processing on the target loss function by taking the new solution as an initial solution, otherwise, not accepting the new solution.
In this embodiment, if the second function value is greater than the first function value, it may be determined whether to accept the perturbation solution according to the Metropolis criterion. Specifically, the metapolis criterion is to calculate the probability of accepting the perturbation solution, i.e., p=exp (- Δt/T), where Δt=second function value-first function value. If the calculated probability is larger than a preset threshold, the disturbance solution is accepted, otherwise, the disturbance solution is not accepted, wherein the preset threshold is a random number in a [0,1 ] interval. In this embodiment, if it is determined according to the metapolis criterion that the disturbance solution selected at this time is not accepted, one disturbance solution is randomly selected again from other disturbance solutions generated based on the disturbance conditions before so as to perform iterative processing on the objective function again. In this embodiment, an iteration number threshold is set for the iteration process of the objective loss function, and after each iteration of the objective loss function and receiving the disturbance solution, it is determined whether the iteration number at that time reaches the iteration number threshold, if so, the disturbance solution received by the iteration is obtained as the calculation resource additional usage after the iteration.
In some embodiments of the present application, if the post-iteration computing resource adding amount still fails to meet the disaster response requirement of the distributed data system, the iterative process may be returned to be performed again until the resource usage rate of the distributed data system after adding the computing resource is lower than the preset resource usage rate threshold. Exemplary, based on a simulated annealing algorithm, reducing the resource utilization rate of the distributed data system according to a preset annealing rate, and resetting the iteration times to zero the iteration times, so that the iteration process is carried out again based on the annealed resource utilization rate of the distributed data system. In this embodiment, the annealing rate λ is set to be a positive number smaller than 1, specifically, a value is between 0.8 and 0.99, and the resource usage rate of the distributed data system is slowly reduced by the annealing rate, where a specific calculation formula is as follows: t (n+1) =λt (n), n=1, 2,3. Where T (n) represents the resource usage of the distributed data system before annealing and T (n+1) represents the resource usage of the distributed data system after annealing. When judging that the resource utilization rate of the distributed data system after adding the computing resource is not lower than a preset resource utilization rate threshold, cooling the distributed data system according to the preset annealing rate, and then returning to iterate again until the resource utilization rate of the distributed data system after adding the computing resource is lower than the preset resource utilization rate threshold, so as to obtain the iterative computing resource adding amount.
It should be understood that, the sequence number of each step in the foregoing embodiment does not mean the execution sequence, and the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way.
In some embodiments of the present application, referring to fig. 6, fig. 6 is a basic block diagram of a disaster response processing device of a distributed data system according to an embodiment of the present application. The apparatus in this embodiment includes units for performing the steps in the method embodiments described above. Refer to the related description in the above method embodiment. For convenience of explanation, only the portions related to the present embodiment are shown. As shown in fig. 6, the disaster response processing device of the distributed data system includes: a judging module 61, a calculating module 62 and a scheduling module 63. Wherein: the judging module 61 is configured to obtain a current resource usage rate of the distributed data system, and judge whether the current resource usage rate of the distributed data system reaches a disaster response starting threshold. The calculation module 62 is configured to solve, if the resource usage rate reaches the disaster response starting threshold, a preset disaster response model to obtain an amount of calculation resources that need to be added for system disaster response. The scheduling module 63 is configured to perform resource scheduling processing on the distributed data system according to the usage of the computing resource to be added, so that the resource usage rate of the distributed data system is reduced to a reference value representing safe operation of the system.
It should be understood that the disaster response processing device of the distributed data system corresponds to the disaster response processing method of the distributed data system one by one, and will not be described herein.
In some embodiments of the present application, referring to fig. 7, fig. 7 is a basic block diagram of an electronic device according to an embodiment of the present application. As shown in fig. 7, the electronic device 7 of this embodiment includes: a processor 71, a memory 72 and a computer program 73 stored in said memory 72 and executable on said processor 71, for example a program of a disaster response handling method of a distributed data system. Processor 71, when executing computer program 73, implements the steps described above in various embodiments of the disaster recovery processing method for each of the distributed data systems. Alternatively, the processor 71 may implement the functions of each module in the embodiments corresponding to the disaster response processing device of the distributed data system when executing the computer program 73. Please refer to the related description in the embodiments, which is not repeated here.
By way of example, the computer program 73 may be divided into one or more modules (units) that are stored in the memory 72 and executed by the processor 71 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing the specified functions for describing the execution of the computer program 73 in the electronic device 7. For example, the computer program 73 may be divided into a judgment module, a calculation module, and a scheduling module, each module having a specific function as described above.
The electronic device may include, but is not limited to, a processor 71, a memory 72. It will be appreciated by those skilled in the art that fig. 7 is merely an example of the electronic device 7 and is not meant to be limiting as the electronic device 7 may include more or fewer components than shown, or may combine certain components, or different components, e.g., the electronic device may further include an input-output device, a network access device, a bus, etc.
The processor 71 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 72 may be an internal storage unit of the electronic device 7, such as a hard disk or a memory of the electronic device 7. The memory 72 may be an external storage device of the electronic device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the electronic device 7. Further, the memory 72 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 72 is used to store the computer program as well as other programs and data required by the electronic device. The memory 72 may also be used to temporarily store data that has been output or is to be output.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps that may implement the various method embodiments described above. In this embodiment, the computer-readable storage medium may be nonvolatile or may be volatile.
Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that may be performed in the various method embodiments described above.
It is appreciated that embodiments of the present application may acquire and process relevant data based on artificial intelligence techniques. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.
It is to be understood that the embodiments of the present application are operational with numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. Embodiments of the application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each method embodiment described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium may include content that is subject to appropriate increases and decreases as required by jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is not included as electrical carrier signals and telecommunication signals.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (6)
1. The disaster response processing method for the distributed data system is characterized by comprising the following steps:
acquiring the current resource utilization rate of a distributed data system, and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold;
if the resource utilization rate reaches the disaster response starting threshold, solving the consumption of the computing resource which needs to be added in the disaster response of the system by adopting a preset disaster response model, wherein the method comprises the following steps: the preset initialized calculation resource adding amount is used as an initial solution of a target loss function in the preset disaster response model to calculate the target loss function, and a first function value is obtained; carrying out disturbance processing on the initial solution according to preset disturbance conditions to obtain a plurality of disturbance solutions corresponding to the initial solution; randomly selecting one disturbance solution from the disturbance solutions as a new solution of a target loss function in the preset disaster response model to calculate the target loss function, and obtaining a second function value; comparing the first function value with the second function value, if the second function value is smaller than or equal to the first function value, receiving the new solution and carrying out next round of iteration processing on the target loss function by taking the new solution as an initial solution until the iteration times reach a preset time threshold value, and obtaining the calculated resource additional consumption after iteration; according to the iterative calculation resource adding amount, simulating and calculating the resource utilization rate of the distributed data system after adding the calculation resource; judging whether the resource utilization rate of the distributed data system after adding the computing resource is lower than a preset resource utilization rate threshold value, if yes, configuring the iterative computing resource adding amount as the computing resource amount which needs to be added by the system according to disaster, otherwise, reducing the resource utilization rate of the distributed data system according to a preset annealing rate and resetting the iteration times so as to carry out iterative processing again based on the annealed resource utilization rate of the distributed data system until the resource utilization rate of the distributed data system after adding the computing resource is lower than the preset resource utilization rate threshold value;
performing resource scheduling processing on the distributed data system according to the consumption of the computing resource to be added, so that the resource utilization rate of the distributed data system is reduced to a reference value representing the safe operation of the system, wherein the mode of performing the resource scheduling processing comprises internal scheduling processing and external access processing, and the method comprises the following steps: the internal scheduling process is resource scheduling between slave nodes in the distributed data system, and the external access process is to access computing resources from outside of the distributed data system.
2. The disaster response processing method of a distributed data system according to claim 1, wherein in the step of performing a disturbance process on the initial solution according to a preset disturbance condition to obtain a plurality of disturbance solutions corresponding to the initial solution, the preset disturbance condition is configured to:
and->
Where ω is denoted as the initial solution and ω' is denoted as the disturbance solution.
3. The disaster recovery processing method of claim 1, wherein after the step of comparing the first function value and the second function value, if the second function value is greater than the first function value, further comprising:
calculating a probability value for accepting the new solution according to a Metropolis criterion according to the first function value and the second function value;
and comparing the calculated probability value with a preset probability threshold, if the probability value is larger than the preset probability threshold, accepting the new solution, and carrying out next round of iterative processing on the target loss function by taking the new solution as an initial solution, otherwise, not accepting the new solution.
4. A disaster response processing device for a distributed data system, comprising:
the judging module is used for acquiring the current resource utilization rate of the distributed data system and judging whether the current resource utilization rate of the distributed data system reaches a disaster response starting threshold value or not;
the calculation module is used for solving the consumption of the calculation resources which need to be added in response to the disaster of the system by adopting a preset disaster response model if the utilization rate of the resources reaches a disaster response starting threshold, and comprises the following steps: the preset initialized calculation resource adding amount is used as an initial solution of a target loss function in the preset disaster response model to calculate the target loss function, and a first function value is obtained; carrying out disturbance processing on the initial solution according to preset disturbance conditions to obtain a plurality of disturbance solutions corresponding to the initial solution; randomly selecting one disturbance solution from the disturbance solutions as a new solution of a target loss function in the preset disaster response model to calculate the target loss function, and obtaining a second function value; comparing the first function value with the second function value, if the second function value is smaller than or equal to the first function value, receiving the new solution and carrying out next round of iteration processing on the target loss function by taking the new solution as an initial solution until the iteration times reach a preset time threshold value, and obtaining the calculated resource additional consumption after iteration; according to the iterative calculation resource adding amount, simulating and calculating the resource utilization rate of the distributed data system after adding the calculation resource; judging whether the resource utilization rate of the distributed data system after adding the computing resource is lower than a preset resource utilization rate threshold value, if yes, configuring the iterative computing resource adding amount as the computing resource amount which needs to be added by the system according to disaster, otherwise, reducing the resource utilization rate of the distributed data system according to a preset annealing rate and resetting the iteration times so as to carry out iterative processing again based on the annealed resource utilization rate of the distributed data system until the resource utilization rate of the distributed data system after adding the computing resource is lower than the preset resource utilization rate threshold value;
the scheduling module is used for performing resource scheduling processing on the distributed data system according to the consumption of the computing resources to be added, so that the resource utilization rate of the distributed data system is reduced to a reference value representing the safe operation of the system, and the mode of performing the resource scheduling processing comprises internal scheduling processing and external access processing, wherein: the internal scheduling process is resource scheduling between slave nodes in the distributed data system, and the external access process is to access computing resources from outside of the distributed data system.
5. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 3 when the computer program is executed.
6. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111007167.7A CN113722292B (en) | 2021-08-30 | 2021-08-30 | Disaster response processing method, device, equipment and storage medium of distributed data system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111007167.7A CN113722292B (en) | 2021-08-30 | 2021-08-30 | Disaster response processing method, device, equipment and storage medium of distributed data system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113722292A CN113722292A (en) | 2021-11-30 |
| CN113722292B true CN113722292B (en) | 2024-02-09 |
Family
ID=78679384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111007167.7A Active CN113722292B (en) | 2021-08-30 | 2021-08-30 | Disaster response processing method, device, equipment and storage medium of distributed data system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113722292B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005263447A (en) * | 2004-03-19 | 2005-09-29 | Yaskawa Electric Corp | Method for scheduling optimum car allocation, and program storage medium thereof |
| CN110515704A (en) * | 2019-08-30 | 2019-11-29 | 广东浪潮大数据研究有限公司 | Resource regulating method and device based on Kubernetes system |
| CN111176753A (en) * | 2019-12-20 | 2020-05-19 | 贝壳技术有限公司 | Cloud resource optimal configuration method and device, electronic equipment and readable storage medium |
| CN111586762A (en) * | 2020-04-29 | 2020-08-25 | 重庆邮电大学 | Task unloading and resource allocation joint optimization method based on edge cooperation |
| WO2020169182A1 (en) * | 2019-02-19 | 2020-08-27 | Nokia Solutions And Networks Oy | Method and apparatus for allocating tasks |
| CN111611274A (en) * | 2020-05-28 | 2020-09-01 | 华中科技大学 | Database query optimization method and system |
| CN112965813A (en) * | 2021-02-10 | 2021-06-15 | 山东英信计算机技术有限公司 | AI platform resource regulation and control method, system and medium |
-
2021
- 2021-08-30 CN CN202111007167.7A patent/CN113722292B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005263447A (en) * | 2004-03-19 | 2005-09-29 | Yaskawa Electric Corp | Method for scheduling optimum car allocation, and program storage medium thereof |
| WO2020169182A1 (en) * | 2019-02-19 | 2020-08-27 | Nokia Solutions And Networks Oy | Method and apparatus for allocating tasks |
| CN110515704A (en) * | 2019-08-30 | 2019-11-29 | 广东浪潮大数据研究有限公司 | Resource regulating method and device based on Kubernetes system |
| CN111176753A (en) * | 2019-12-20 | 2020-05-19 | 贝壳技术有限公司 | Cloud resource optimal configuration method and device, electronic equipment and readable storage medium |
| CN111586762A (en) * | 2020-04-29 | 2020-08-25 | 重庆邮电大学 | Task unloading and resource allocation joint optimization method based on edge cooperation |
| CN111611274A (en) * | 2020-05-28 | 2020-09-01 | 华中科技大学 | Database query optimization method and system |
| CN112965813A (en) * | 2021-02-10 | 2021-06-15 | 山东英信计算机技术有限公司 | AI platform resource regulation and control method, system and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113722292A (en) | 2021-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3447642B1 (en) | System and method for predicting application performance for large data size on big data cluster | |
| CN113420073A (en) | Abnormal sample detection method based on improved isolated forest and related equipment | |
| CN111159897B (en) | Target optimization method and device based on system modeling application | |
| CN104426696A (en) | Fault processing method and device | |
| CN113537337A (en) | Training method, abnormality detection method, apparatus, device, and storage medium | |
| CN112783508B (en) | File compiling method, device, equipment and storage medium | |
| CN114780338A (en) | Host information processing method and device, electronic equipment and computer readable medium | |
| CN111339072A (en) | User behavior based change value analysis method and device, electronic device and medium | |
| CN113722292B (en) | Disaster response processing method, device, equipment and storage medium of distributed data system | |
| EP3855316A1 (en) | Optimizing breakeven points for enhancing system performance | |
| CN113495825A (en) | Line alarm processing method and device, electronic equipment and readable storage medium | |
| CN113641670B (en) | Data storage and data retrieval method and device, electronic equipment and storage medium | |
| CN116662001A (en) | Event processing method and device | |
| CN116166427A (en) | Automatic capacity expansion and contraction method, device, equipment and storage medium | |
| CN115564329A (en) | Typical capacity scene determining method, device, equipment and storage medium | |
| CN115495151A (en) | Rule engine migration method, device, equipment, storage medium and program product | |
| CN114676272A (en) | Information processing method, device and equipment of multimedia resource and storage medium | |
| CN112764957A (en) | Application fault delimiting method and device | |
| CN113343064B (en) | Data processing method, apparatus, device, storage medium, and computer program product | |
| CN110955521A (en) | Power distribution network wide area distributed type sinking calculation system and method based on big data | |
| CN115081233B (en) | Flow simulation method and electronic equipment | |
| CN117974009B (en) | Method and device for determining task splitting rate, electronic equipment and storage medium | |
| CN115858324B (en) | AI-based IT equipment fault processing method, apparatus, equipment and medium | |
| CN109753405B (en) | Application resource consumption detection method and device, storage medium and electronic equipment | |
| US20240004765A1 (en) | Data processing method and apparatus for distributed storage system, device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220928 Address after: Room 2601 (Unit 07), Qianhai Free Trade Building, No. 3048, Xinghai Avenue, Nanshan Street, Qianhai Shenzhen-Hong Kong Cooperation Zone, Shenzhen, Guangdong 518000 Applicant after: Shenzhen Ping An Smart Healthcare Technology Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |