CN112650737A - Data processing method, data processing apparatus, storage medium, and electronic device - Google Patents
Data processing method, data processing apparatus, storage medium, and electronic device Download PDFInfo
- Publication number
- CN112650737A CN112650737A CN202011627012.9A CN202011627012A CN112650737A CN 112650737 A CN112650737 A CN 112650737A CN 202011627012 A CN202011627012 A CN 202011627012A CN 112650737 A CN112650737 A CN 112650737A
- Authority
- CN
- China
- Prior art keywords
- slot
- slot position
- determining
- node
- evaluation value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 15
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000011156 evaluation Methods 0.000 claims abstract description 45
- 230000005012 migration Effects 0.000 claims abstract description 23
- 238000013508 migration Methods 0.000 claims abstract description 23
- 238000013500 data storage Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 230000002159 abnormal effect Effects 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a data processing device, a storage medium and electronic equipment. The embodiment of the invention determines the slot position information of the slot position serving as the data storage range in the target database, and determines the evaluation value of each slot position according to the slot position information of each slot position, so that the slot position migration is carried out according to the evaluation value of each slot position, and the load of at least one slot position is balanced. Load balancing among the slots can balance the load of each node in the database cluster, so that the embodiment of the invention can balance the load among the databases in the database cluster, thereby effectively reducing the access pressure of a certain database.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, a data processing apparatus, a storage medium, and an electronic device.
Background
With the rapid development of computer technology, the size of data and the access amount of data are continuously increasing. To accommodate the increasing volume of data access, more and more technicians choose to share the access pressure of a single database by deploying a cluster of databases. But load imbalance conditions tend to occur between databases, thus causing greater access pressure to at least one database in a database cluster.
Disclosure of Invention
In view of the above, an object of the embodiments of the present invention is to provide a data processing method, a data processing apparatus, a storage medium, and an electronic device, which are used for balancing loads among databases in a database cluster, so as to effectively reduce the access pressure of a certain database.
According to a first aspect of embodiments of the present invention, there is provided a data processing method, the method including:
determining slot position information of each slot position in a target database cluster, wherein the slot position is a data storage range;
determining an evaluation value of a corresponding slot position according to the slot position information;
and carrying out slot position migration according to the evaluation value of each slot position so as to balance the load of at least one slot position.
Preferably, the slot information includes a slot request quantity, a slot memory usage quantity, and a resource consumption index of the corresponding node of each slot.
Preferably, the determining slot position information of each slot position in the target database cluster includes:
acquiring a data query instruction received by an access layer;
determining a slot position corresponding to each data query instruction;
and regarding each slot, taking the number of the corresponding data query instructions as the slot request number.
Preferably, the determining slot position information of each slot position in the target database cluster includes:
acquiring a memory mirror image file of the target database cluster;
analyzing the memory mirror image file and determining key value pair data in the memory mirror image file;
determining a slot position corresponding to each key value pair data;
and regarding each slot position, taking the sum of the memory usage of the corresponding key value pair data as the memory usage of the slot position.
Preferably, the resource consumption index includes instruction response time consumption of the corresponding node, node request quantity, node memory usage and node processor usage;
the determining slot position information of each slot position in the target database cluster comprises:
and acquiring the instruction response time, the node request quantity, the node memory usage and the node processor usage by a preset distributed system monitor.
Preferably, the determining an evaluation value of a corresponding slot from each of the slot information includes:
and respectively calculating the weighted sum of the slot position information as the evaluation value of the corresponding slot point.
Preferably, the performing slot migration according to the evaluation value of each slot includes:
determining a slot position to be migrated and a target slot position according to the evaluation values in an iterative mode, and migrating the slot position to be migrated to the target slot position until a preset balance condition is met, wherein the slot position to be migrated is a slot position of which the evaluation value meets a first condition, and the target slot position is a slot position of which the evaluation value meets a second condition.
According to a second aspect of embodiments of the present invention, there is provided a data processing apparatus, the apparatus comprising:
the information determining unit is used for determining slot position information of each slot position in the target database cluster, wherein the slot position is a data storage range;
an evaluation value determining unit, configured to determine an evaluation value of a corresponding slot according to each of the slot position information;
and the slot position migration unit is used for performing slot position migration according to the evaluation value of each slot position so as to balance the load of at least one slot position.
According to a third aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method of any of the first aspects.
According to a fourth aspect of embodiments of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of the first aspect.
The embodiment of the invention determines the slot position information of the slot position serving as the data storage range in the target database, and determines the evaluation value of each slot position according to the slot position information of each slot position, so that the slot position migration is carried out according to the evaluation value of each slot position, and the load of at least one slot position is balanced. Load balancing among the slots can balance the load of each node in the database cluster, so that the embodiment of the invention can balance the load among the databases in the database cluster, thereby effectively reducing the access pressure of a certain database.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a data processing method of a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing method according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of a data processing apparatus according to a second embodiment of the present invention;
fig. 4 is a schematic diagram of an electronic device according to a third embodiment of the invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention.
Further, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.
Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including, but not limited to".
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.
To accommodate the increasing volume of data access, more and more technicians choose to share the access pressure of a single database by deploying a cluster of databases. The database cluster has a plurality of databases, so that the storage capacity of the data is also remarkably stronger than that of a single database. Taking Redis cluster as an example, the Redis cluster includes a plurality of nodes, each node corresponding to a master database or a slave database. In a Redis cluster, the data stored by each node may be mapped to different slots, and each node may correspond to a different number of slots.
Slot (slot) is also a data storage range, and when nodes are added or deleted in the Redis cluster, only the slot needs to be migrated, and the nodes do not need to be added or deleted after the Redis service is stopped, so that the stability of the Redis cluster is improved.
The number of slots for a Redis cluster is typically 16384. After receiving a data query instruction sent by a user, a Redis server in the Redis cluster may parse the data query instruction, determine a key (i.e., an index) in the data query instruction, and then determine a slot corresponding to the key of the data query instruction, specifically, may determine the slot corresponding to the key through a Cyclic Redundancy Check (CRC) algorithm, such as a CRC-8 algorithm, a CRC-16 algorithm, and the like, so as to obtain a query result (i.e., a value) corresponding to the data query instruction through the slot corresponding to the key.
However, the existing Redis cluster does not support the provision of the slot information of each slot, so that when an abnormal condition such as an overload (that is, an excessive access pressure) occurs in a certain slot, the Redis server cannot timely sense the abnormal condition, and the server cannot accurately locate the slot in the abnormal condition, so that the slot migration cannot be performed more accurately, which may aggravate the access pressure of the database corresponding to the slot in the abnormal condition.
Fig. 1 is a flowchart of a data processing method according to a first embodiment of the present invention. As shown in fig. 1, the method of the present embodiment includes the following steps:
and step S100, determining slot position information of each slot position in the target database cluster.
In this embodiment, the slot information is used to measure the load condition of each slot, and may specifically include the slot request quantity, the slot memory usage quantity, and the resource consumption index of the corresponding node of each slot. In this step, the server may determine the slot position information in different ways.
The slot position information is the slot position request quantity. The server can obtain the data query instruction received by the access layer, and determine the slot position corresponding to each data query instruction, and for each slot position, the server can respectively use the number of the data query instruction corresponding to each slot position as the slot position request number of the slot position. The access layer refers to a portion directly accessed for user connection in a network, and may specifically be a Redis access layer client in this embodiment. When the user side sends data query instructions such as get instructions and set instructions to the access layer, the server can obtain the data query instructions, determine keys in the data query instructions, then determine CRC codes corresponding to the keys according to a CRC algorithm, determine slots corresponding to the keys according to the CRC codes, and accordingly obtain values corresponding to the data query instructions according to the corresponding slots.
The CRC algorithm can correct errors in the information transmission process, can complete the calculation of data check codes in a very short time, quickly complete the error correction process, greatly improve the communication speed of a computer in a data packet automatic retransmission mode, and provide guarantee for the communication efficiency and safety.
The CRC16 algorithm is the most common CRC algorithm, so the embodiment chooses to determine the slot corresponding to the data query instruction through the CRC16 algorithm. The CRC16 algorithm is calculated as follows: (1) a preset 16-bit register is used for storing hexadecimal FFFFFF (all 1), and the 16-bit register is also a CRC register; (2) carrying out exclusive-or operation on 8 bits of a first byte in the data frame and a low byte in the CRC register, and storing an operation result in the CRC register; (3) shifting the CRC register to the right by one bit, filling the most significant bit with a number 0, and then detecting the least significant bit; (4) if the lowest bit is 0, repeating the step (3); if the lowest bit is 1, performing exclusive-or operation on the CRC register and a preset value (0A 001H); (5) repeating (3) and (4) until the CRC register is shifted eight times; (6) repeating the steps (2) and (5) until all bytes are completely processed, and obtaining the CRC code of the data frame. The register is used for storing binary codes; the data frame, i.e. the pdu of the data link layer, corresponds to a key in this embodiment.
For example, the key of the data query command is 7E000560313233, and the server may determine, through the CRC16 algorithm (i.e., the above operation step), that the binary CRC code corresponding to the key is 1011110101001000.
After determining the CRC code corresponding to the key, the server may calculate a remainder of the CRC code and the number of the slot positions, thereby determining the slot position corresponding to the data query instruction. For example, the key of the data query instruction corresponds to 1011110101001000 CRC code, the number of slots in the Redis cluster is 16384, the server may convert the binary CRC code corresponding to the key into decimal CRC code, that is, 48456, and determine that the remainder of the decimal CRC code divided by the number of slots is 15688, that is, when the first slot is 0, the 15689 th slot may be determined as the slot corresponding to the data query instruction.
For each slot, the server may determine the number of data query instructions corresponding to each slot, and use the number of data query instructions as the slot request number of the slot.
And (II) the slot position information is the slot position memory usage amount. The server can establish long connection with each node in the Redis cluster through dumptools (unloading tools), and unload the memory mirror image file of each node to the local. Therefore, the server can analyze the memory image file transferred to the local, determine key-value data in the memory image file, and determine a slot corresponding to each key-value data. The memory mirror image file also records the memory usage amount of each key value pair data, so that the server can respectively use the memory usage amount of the key value pair data corresponding to each node as the slot memory usage amount of the slot according to the memory mirror image file.
Similar to the manner of determining the slot request number, the server may also determine a key in each key value pair, then determine a CRC code corresponding to the key according to a CRC algorithm, and determine a slot corresponding to the key according to the CRC code, thereby determining the slot memory usage amount of each slot.
And (III) the slot position information is the resource consumption index of the corresponding node. The resource consumption index may include instruction response time consumption of the corresponding node, node request number, node memory usage amount, and node processor usage rate. The instruction response time is an average value of the time consumed by each node for responding to the data query instruction, and the node processor utilization rate is a CPU (Central Processing Unit) utilization rate of the node.
In this embodiment, a predetermined distributed system monitor may be pre-deployed in the server, and the predetermined distributed system monitor obtains instruction response time, node request number, node memory usage amount, and node processor usage rate of each node.
The predetermined distributed system monitor of this embodiment may be zabbix. Zabbix can monitor various network parameters to ensure the safe operation of the server system. Zabbix consists of two parts, including Zabbix server and Zabbix agent, where Zabbix server is deployed inside the server, so that monitoring of each node can be provided by means of Zabbix agent deployed at each node in the Redis cluster.
And step S200, determining the evaluation value of the corresponding slot position according to the slot position information.
In this step, the server may calculate a weighted sum of each slot position information as an evaluation value of the corresponding slot point, respectively.
Alternatively, the server may obtain slot position information of each slot position, determine a weighted sum of each slot position information according to a preset weight of each item of information, and use the calculated weighted sum as an evaluation value of the corresponding slot position in the same period.
For example, the slot information of the slot S1 includes the slot request number of the slot S1, the slot memory usage amount, and the resource consumption index of the corresponding node, where the slot request number is 100, the memory usage amount is 0.2%, the instruction response time consumption of the corresponding node in the resource consumption index is 3ms, the node request number is 1500, the node memory usage amount is 5%, and the node processor usage rate is 3%, the server may obtain weights corresponding to the information items, and calculate a weighted sum of the information items, so as to determine the weighted sum as the evaluation value of the slot S1.
Optionally, the server may also obtain slot information of each slot in a predetermined number of cycles, and then calculate a weighted sum of the same information in different cycles as the slot information of the corresponding slot.
For example, if the predetermined number is 5 and the number of slot requests for the slot S2 in different periods is 300,350, 480,630 and 1210, the server may obtain a weight corresponding to the number of slot requests in each period and calculate a weighted sum of the numbers of slot requests in different periods, thereby determining the weighted sum as the evaluation value of the slot S2.
In this step, the server may determine the slot position to be migrated and the target slot position according to the evaluation value of each slot position in an iterative manner, and migrate the slot position to be migrated to the target slot position until a predetermined balance condition is satisfied. The slot to be migrated may be a slot whose evaluation value satisfies a first condition, the target slot may be a slot whose evaluation value satisfies a second condition, and the predetermined balance condition may be that a difference value between at least part of information of the slot request quantity, the slot memory usage quantity, and the instruction response time consumption of the corresponding node, the node request quantity, the node memory usage quantity, and the node processor usage rate of each slot is smaller than a first threshold value. The first condition may be set such that the evaluation value is less than (or equal to) the second threshold value and ranks at the top n (n is a predetermined integer of 1 or more) bits of the smallest, and the second condition may be set such that the evaluation value is greater than (or equal to) the third threshold value and ranks at the top m (m is a predetermined integer of 1 or more) bits of the largest.
After the slot position to be migrated and the target slot position are determined, the server may send a cluster setaslot migration instruction to the node where the target slot position is located, change the state of the target slot position into migration (import), send a cluster setaslot migration instruction to the node where the slot position to be migrated is located, change the state of the slot position to be migrated into migration (migration), then control the node where the slot position to be migrated to execute the cluster setkeyaslot instruction, obtain key value data in the slot position to be migrated, and control the node where the slot position to be migrated to migrate the obtained key value to the target node in batch through a pipeline (pipeline) mechanism until all the key value data in the slot position to be migrated are migrated to the target node, so as to achieve the purpose of slot position migration.
It is easily understood that the method of the present embodiment may be performed in a predetermined cycle, that is, the server may perform steps S100 to S300 once in each cycle. Therefore, the server can perform load balancing on each slot position according to a preset period, so that the possibility that a node (namely a database) where a certain slot position is located bears excessive access pressure is avoided.
Fig. 2 is a schematic diagram of a data processing method according to a first embodiment of the present invention. As shown in fig. 2, a node 1-node n is n nodes in a database cluster, an ith node (i is a predetermined integer greater than or equal to 1 and less than or equal to n) includes mn slots, and each node has a zabbix agent (that is, ZA) disposed therein in advance. Therefore, the server can obtain the resource consumption index of each node through the internally deployed zabbix server, and obtain the slot position request quantity and the slot position memory usage quantity of each slot position as the slot position information of each slot position. After the slot position information of each slot position is obtained, the server may calculate an evaluation value of each slot position, determine that the slot position to be migrated is the 1 st slot position in the 1 st node, that is, slot1,1, and determine that the target slot position is the mn th slot position in the nth node, that is, slot n, mn. The server may then perform slot migration to balance the load of at least one slot.
In this embodiment, slot position information of a slot position serving as a data storage range in the target database is determined, and an evaluation value of each slot position is determined according to the slot position information of each slot position, so that slot position migration is performed according to the evaluation value of each slot position, and a load of at least one slot position is balanced. Load balancing among slots can balance loads of all nodes in the database cluster, so that the load can be balanced among all databases in the database cluster, and the access pressure of a certain database is effectively reduced.
Fig. 3 is a schematic diagram of a data processing apparatus according to a second embodiment of the present invention. As shown in fig. 3, the apparatus of the present embodiment includes an information determining unit 31, an evaluation value determining unit 32, and a slot migrating unit 33.
The information determining unit 31 is configured to determine slot position information of each slot position in the target database cluster, where the slot position is a data storage range. The evaluation value determining unit 32 is configured to determine an evaluation value of the corresponding slot from each of the slot information. The slot migration unit 33 is configured to perform slot migration according to the evaluation value of each slot, so as to balance a load of at least one slot.
In this embodiment, slot position information of a slot position serving as a data storage range in the target database is determined, and an evaluation value of each slot position is determined according to the slot position information of each slot position, so that slot position migration is performed according to the evaluation value of each slot position, and a load of at least one slot position is balanced. Load balancing among slots can balance loads of all nodes in the database cluster, so that the load can be balanced among all databases in the database cluster, and the access pressure of a certain database is effectively reduced.
Fig. 4 is a schematic diagram of an electronic device according to a third embodiment of the invention. The electronic device shown in fig. 4 is a general-purpose data processing apparatus, and may be specifically a first terminal, a second terminal or a server according to an embodiment of the present invention, and includes a general-purpose computer hardware structure, which includes at least a processor 41 and a memory 42. The processor 41 and the memory 42 are connected by a bus 43. The memory 42 is adapted to store instructions or programs executable by the processor 41. Processor 41 may be a stand-alone microprocessor or may be a collection of one or more microprocessors. Thus, processor 41 implements the processing of data and the control of other devices by executing commands stored in memory 42 to thereby execute the method flows of embodiments of the present invention as described above. The bus 43 connects the above components together, and also connects the above components to a display controller 44 and a display device and an input/output (I/O) device 45. Input/output (I/O) devices 45 may be a mouse, keyboard, modem, network interface, touch input device, motion sensing input device, printer, and other devices known in the art. Typically, an input/output (I/O) device 45 is connected to the system through an input/output (I/O) controller 46.
The memory 42 may store, among other things, software components such as an operating system, communication modules, interaction modules, and application programs. Each of the modules and applications described above corresponds to a set of executable program instructions that perform one or more functions and methods described in embodiments of the invention.
The flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention described above illustrate various aspects of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
Also, as will be appreciated by one skilled in the art, aspects of embodiments of the present invention may be embodied as a system, method or computer program product. Accordingly, various aspects of embodiments of the invention may take the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," module "or" system. Further, aspects of the invention may take the form of: a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.
Any combination of one or more computer-readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of embodiments of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to: electromagnetic, optical, or any suitable combination thereof. The computer readable signal medium may be any of the following computer readable media: is not a computer readable storage medium and may communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including: object oriented programming languages such as Java, Smalltalk, C + +, PHP, Python, and the like; and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package; executing in part on a user computer and in part on a remote computer; or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
determining slot position information of each slot position in a target database cluster, wherein the slot position is a data storage range;
determining an evaluation value of a corresponding slot position according to the slot position information;
and carrying out slot position migration according to the evaluation value of each slot position so as to balance the load of at least one slot position.
2. The method of claim 1, wherein the slot information comprises a slot request number, a slot memory usage amount, and a resource consumption indicator for a corresponding node for each slot.
3. The method of claim 2, wherein determining slot information for each slot in a target database cluster comprises:
acquiring a data query instruction received by an access layer;
determining a slot position corresponding to each data query instruction;
and regarding each slot, taking the number of the corresponding data query instructions as the slot request number.
4. The method of claim 2, wherein determining slot information for each slot in a target database cluster comprises:
acquiring a memory mirror image file of the target database cluster;
analyzing the memory mirror image file and determining key value pair data in the memory mirror image file;
determining a slot position corresponding to each key value pair data;
and regarding each slot position, taking the sum of the memory usage of the corresponding key value pair data as the memory usage of the slot position.
5. The method of claim 2, wherein the resource consumption metrics include instruction response time consumption, node request amount, node memory usage amount, and node processor usage rate of the corresponding node;
the determining slot position information of each slot position in the target database cluster comprises:
and acquiring the instruction response time, the node request quantity, the node memory usage and the node processor usage by a preset distributed system monitor.
6. The method of claim 1, wherein said determining an evaluation value for a corresponding slot from each of said slot information comprises:
and respectively calculating the weighted sum of the slot position information as the evaluation value of the corresponding slot point.
7. The method of claim 1, wherein said performing slot migration based on said evaluated value for each of said slots comprises:
determining a slot position to be migrated and a target slot position according to the evaluation values in an iterative mode, and migrating the slot position to be migrated to the target slot position until a preset balance condition is met, wherein the slot position to be migrated is a slot position of which the evaluation value meets a first condition, and the target slot position is a slot position of which the evaluation value meets a second condition.
8. A data processing apparatus, characterized in that the apparatus comprises:
the information determining unit is used for determining slot position information of each slot position in the target database cluster, wherein the slot position is a data storage range;
an evaluation value determining unit, configured to determine an evaluation value of a corresponding slot according to each of the slot position information;
and the slot position migration unit is used for performing slot position migration according to the evaluation value of each slot position so as to balance the load of at least one slot position.
9. A computer-readable storage medium on which computer program instructions are stored, which, when executed by a processor, implement the method of any one of claims 1-7.
10. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627012.9A CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627012.9A CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112650737A true CN112650737A (en) | 2021-04-13 |
| CN112650737B CN112650737B (en) | 2024-03-19 |
Family
ID=75367347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011627012.9A Active CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650737B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117439993A (en) * | 2023-12-22 | 2024-01-23 | 中电云计算技术有限公司 | Redis cluster load balancing method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2633413A2 (en) * | 2010-10-20 | 2013-09-04 | Microsoft Corporation | Low ram space, high-throughput persistent key-value store using secondary memory |
| CN105516367A (en) * | 2016-02-02 | 2016-04-20 | 北京百度网讯科技有限公司 | Distributed data storage system, method and device |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
-
2020
- 2020-12-31 CN CN202011627012.9A patent/CN112650737B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2633413A2 (en) * | 2010-10-20 | 2013-09-04 | Microsoft Corporation | Low ram space, high-throughput persistent key-value store using secondary memory |
| CN105516367A (en) * | 2016-02-02 | 2016-04-20 | 北京百度网讯科技有限公司 | Distributed data storage system, method and device |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
Non-Patent Citations (1)
| Title |
|---|
| 王永亮;: "基于二次映射的哈希负载均衡方法", 信息记录材料, no. 12, 1 December 2018 (2018-12-01) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117439993A (en) * | 2023-12-22 | 2024-01-23 | 中电云计算技术有限公司 | Redis cluster load balancing method, device, equipment and storage medium |
| CN117439993B (en) * | 2023-12-22 | 2024-04-16 | 中电云计算技术有限公司 | Redis cluster load balancing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112650737B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11204827B2 (en) | Using a machine learning module to determine when to perform error checking of a storage unit | |
| US10909018B2 (en) | System and method for end-to-end application root cause recommendation | |
| CN109976975B (en) | Disk capacity prediction method and device, electronic equipment and storage medium | |
| CN112231187B (en) | Micro-service abnormity analysis method and device | |
| CN104756084A (en) | Operation management device and operation management method | |
| CN112380046B (en) | Calculation result verification method, system, device, equipment and storage medium | |
| CN112650737B (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN110187995A (en) | A kind of method and device for fusing of the peer node that fuses | |
| CN115269108A (en) | Data processing method, device and equipment | |
| CN112860512B (en) | Interface monitoring optimization method and device, computer equipment and storage medium | |
| CN113342517A (en) | Resource request forwarding method and device, electronic equipment and readable storage medium | |
| KR102645576B1 (en) | Method of simulating quantum computing system and quantum computing simulation system | |
| CN112671602A (en) | Data processing method, device, system, equipment and storage medium of edge node | |
| US10387578B1 (en) | Utilization limiting for nested object queries | |
| CN112331249A (en) | Method and device for predicting service life of storage device, terminal equipment and storage medium | |
| US20130253899A1 (en) | Determining Simulation Fidelity in a Self-Optimized Simulation of a Complex System | |
| CN103049665B (en) | A kind of multivariate computing method of the configuration computational logic based on variable and device | |
| US20120041741A1 (en) | Dynamically Adjusting Simulation Fidelity in a Self-Optimized Simulation of a Complex System | |
| CN115129509A (en) | Data transmission method, device and medium | |
| CN112085589B (en) | Method and device for determining safety of rule model and server | |
| CN115081429A (en) | Instruction checking method, device, equipment and storage medium | |
| CN108512711B (en) | Method, device and equipment for obtaining instantaneous interface access statistic | |
| CN111641959A (en) | Method and device for detecting network environment information | |
| US20200358679A1 (en) | Reducing data storage and network traffic with data compression | |
| US8437993B2 (en) | Dynamically adjusting simulation fidelity based on changes in activity levels of components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |