CN112650737B - Data processing method, data processing device, storage medium and electronic equipment - Google Patents
Data processing method, data processing device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112650737B CN112650737B CN202011627012.9A CN202011627012A CN112650737B CN 112650737 B CN112650737 B CN 112650737B CN 202011627012 A CN202011627012 A CN 202011627012A CN 112650737 B CN112650737 B CN 112650737B
- Authority
- CN
- China
- Prior art keywords
- slot
- determining
- evaluation value
- node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 15
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000011156 evaluation Methods 0.000 claims abstract description 47
- 230000005012 migration Effects 0.000 claims abstract description 19
- 238000013508 migration Methods 0.000 claims abstract description 19
- 238000013500 data storage Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 14
- 230000004044 response Effects 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000002159 abnormal effect Effects 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a data processing device, a storage medium and electronic equipment. The embodiment of the invention determines the slot position information of the slot positions serving as the data storage range in the target database, and determines the evaluation value of each slot position according to the slot position information of each slot position, so that the slot position migration is performed according to the evaluation value of each slot position to balance the load of at least one slot position. The load balancing among the slots can lead the load of each node in the database cluster to be balanced, so the embodiment of the invention can balance the load among all databases in the database cluster, thereby effectively reducing the access pressure of a certain database.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, a data processing apparatus, a storage medium, and an electronic device.
Background
With the rapid development of computer technology, the size of data and the data access volume are also increasing. To accommodate the increasing amount of data access, more and more technicians choose to share the access pressure of a single database by deploying a database cluster. However, the load imbalance between databases is liable to occur, and thus a large access pressure is applied to at least one database in the database cluster.
Disclosure of Invention
In view of the foregoing, an object of an embodiment of the present invention is to provide a data processing method, a data processing apparatus, a storage medium, and an electronic device, for balancing loads among databases in a database cluster, so as to effectively reduce access pressure of a certain database.
According to a first aspect of an embodiment of the present invention, there is provided a data processing method, the method including:
determining slot position information of each slot position in a target database cluster, wherein the slot position is a data storage range;
determining an evaluation value of the corresponding slot according to the slot information;
and carrying out slot migration according to the evaluation value of each slot so as to balance the load of at least one slot.
Preferably, the slot information includes a slot request number of each slot, a slot memory usage amount, and a resource consumption index of a corresponding node.
Preferably, the determining the slot information of each slot in the target database cluster includes:
acquiring a data query instruction received by an access layer;
determining a slot position corresponding to each data query instruction;
and for each slot, taking the corresponding number of the data query instructions as the slot request number.
Preferably, the determining the slot information of each slot in the target database cluster includes:
acquiring a memory mirror image file of the target database cluster;
analyzing the memory mirror image file and determining key value pair data in the memory mirror image file;
determining the corresponding slot position of each key value to the data;
and for each slot, taking the sum of the memory usage of the corresponding key value pair data as the memory usage of the slot.
Preferably, the resource consumption index comprises instruction response time consumption, node request quantity, node memory usage and node processor usage of the corresponding node;
the determining the slot position information of each slot position in the target database cluster comprises the following steps:
and acquiring the instruction response time consumption, the node request quantity, the node memory usage and the node processor usage through a preset distributed system monitor.
Preferably, the determining the evaluation value of the corresponding slot according to each slot information includes:
and respectively calculating a weighted sum of the slot information as the evaluation value of the corresponding slot point.
Preferably, the performing slot migration according to the evaluation value of each slot includes:
and determining a to-be-migrated slot and a target slot according to each evaluation value in an iterative mode, and migrating the to-be-migrated slot to the target slot until a preset balance condition is met, wherein the to-be-migrated slot is a slot with the evaluation value meeting a first condition, and the target slot is a slot with the evaluation value meeting a second condition.
According to a second aspect of embodiments of the present invention, there is provided a data processing apparatus, the apparatus comprising:
the information determining unit is used for determining slot position information of each slot position in the target database cluster, wherein the slot position is a data storage range;
an evaluation value determining unit for determining an evaluation value of a corresponding slot according to each slot information;
and the slot migration unit is used for carrying out slot migration according to the evaluation value of each slot so as to balance the load of at least one slot.
According to a third aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method according to any of the first aspects.
According to a fourth aspect of embodiments of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the memory is for storing one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as in any of the first aspects.
The embodiment of the invention determines the slot position information of the slot positions serving as the data storage range in the target database, and determines the evaluation value of each slot position according to the slot position information of each slot position, so that the slot position migration is performed according to the evaluation value of each slot position to balance the load of at least one slot position. The load balancing among the slots can lead the load of each node in the database cluster to be balanced, so the embodiment of the invention can balance the load among all databases in the database cluster, thereby effectively reducing the access pressure of a certain database.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a data processing method of a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing method according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of a data processing apparatus according to a second embodiment of the present invention;
fig. 4 is a schematic view of an electronic device according to a third embodiment of the present invention.
Detailed Description
The present invention is described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth in detail. The present invention will be fully understood by those skilled in the art without the details described herein. Well-known methods, procedures, flows, components and circuits have not been described in detail so as not to obscure the nature of the invention.
Moreover, those of ordinary skill in the art will appreciate that the drawings are provided herein for illustrative purposes and that the drawings are not necessarily drawn to scale.
Unless the context clearly requires otherwise, the words "comprise," "comprising," and the like in the description are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".
In the description of the present invention, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.
To accommodate the increasing amount of data access, more and more technicians choose to share the access pressure of a single database by deploying a database cluster. The database cluster has a plurality of databases, so the storage capacity of the data is significantly stronger than that of a single database. Taking Redis cluster as an example, redis cluster includes a plurality of nodes, each node corresponding to one master database or slave database. In a Redis cluster, data stored by each node may be mapped onto different slots, and each node may correspond to a different number of slots.
The slot (slot) is the data storage range, and when the node is added or deleted in the Redis cluster, the slot is only required to be migrated, and the node is not required to be added or deleted after the Redis service is stopped, so that the stability of the Redis cluster is improved.
The number of slots of the Redis cluster is typically 16384. After receiving the data query instruction sent by the user end, the Redis server in the Redis cluster can analyze the data query instruction, determine a key (i.e. an index) in the data query instruction, then determine a slot corresponding to the key of the data query instruction, and specifically determine the slot corresponding to the key through a CRC (Cyclic Redundancy Check) algorithm, such as a CRC-8 algorithm, a CRC-16 algorithm, and the like, so as to obtain a query result (i.e. a value) corresponding to the data query instruction through the slot corresponding to the key.
However, the existing Redis cluster does not support providing slot information of each slot, so when an abnormal condition such as overload (i.e., excessive access pressure) occurs in a certain slot, the Redis server cannot timely sense the abnormal condition, and the server cannot accurately locate the slot in which the abnormal condition occurs, so that the slot migration cannot be accurately performed, and the access pressure of the database corresponding to the slot in which the abnormal condition occurs is increased.
Fig. 1 is a flowchart of a data processing method according to a first embodiment of the present invention. As shown in fig. 1, the method of the present embodiment includes the following steps:
step S100, determining slot position information of each slot position in the target database cluster.
In this embodiment, the slot information is used to measure the load condition of each slot, and may specifically include the number of slot requests of each slot, the usage of the slot memory, and the resource consumption index of the corresponding node. In this step, the server may determine in different ways for different slot information.
The slot information is the number of slot requests. The server can acquire the data query instructions received by the access layer, determine the slots corresponding to the data query instructions, and for each slot, the server can respectively use the number of the data query instructions corresponding to each slot as the slot request number of the slot. The access layer refers to a portion of the network that directly faces to a user connection to obtain access, and in this embodiment, may be specifically a Redis access layer client. When the user side sends data query instructions such as get instructions and set instructions to the access layer, the server can acquire the data query instructions, determine keys in the data query instructions, then determine CRC codes corresponding to the keys according to a CRC algorithm, and determine slots corresponding to the keys according to the CRC codes, so that values corresponding to the data query instructions are acquired according to the corresponding slots.
The CRC algorithm can correct errors in the information transmission process, can complete calculation of the data check code in extremely short time, rapidly complete the error correction process, greatly improve the communication speed of a computer in a data packet automatic retransmission mode, and provide guarantee for communication efficiency and safety.
The CRC16 algorithm is the most common CRC algorithm, so this embodiment selects the slots corresponding to the data query instruction determined by the CRC16 algorithm. The calculation process of the CRC16 algorithm is as follows: (1) The preset 16-bit register is used for storing hexadecimal FFFFs (all 1), and the 16-bit register is a CRC register; (2) Performing exclusive OR operation on 8 bits of the first byte in the data frame and the low byte in the CRC register, and storing an operation result in the CRC register; (3) Shifting the CRC register by one bit to the right, filling the digit 0 in the most significant bit, and then detecting the least significant bit; (4) repeating (3) if the lowest bit is 0; if the lowest bit is 1, performing exclusive OR operation on the CRC register and a preset value (0A 001H); (5) Repeating (3) and (4) until eight shifts are performed on the CRC register; (6) Repeating (2) - (5) until all bytes are processed completely, and obtaining CRC coding of the data frame. Wherein the register is used for storing binary codes; the data frame, i.e. the protocol data unit of the data link layer, corresponds in this embodiment to a key.
For example, the key of the data query instruction is 7E000560313233, and the server can determine that the binary CRC code corresponding to the key is 1011110101001000 through the CRC16 algorithm (i.e., the above operation step).
After determining the CRC code corresponding to the key, the server may calculate the remainder of the CRC code and the number of slots, thereby determining the slots corresponding to the data query instruction. For example, the CRC code corresponding to the key of the data query instruction is 1011110101001000, the number of slots in the dis cluster is 16384, the server may convert the binary CRC code corresponding to the key into a decimal CRC code, that is, 48456, and determine the remainder of dividing the decimal CRC code by the number of slots to be 15688, that is, when the first slot is 0, the 15689 th slot may be determined as the slot corresponding to the data query instruction.
For each slot, the server may determine the number of data query instructions corresponding to each slot, and use the number of data query instructions as the number of slot requests for the slot.
And (II) the slot information is the slot memory usage. The server can establish long connection with each node in the Redis cluster through dumptools (dump tool), and dump the memory image file of each node to the local. Therefore, the server can analyze the memory mirror image file transferred to the local, determine key-value pair (key-value) data in the memory mirror image file, and determine the slot positions corresponding to the data by each key value. The memory mirror file also records the memory usage of each key value to the data, so the server can respectively use the sum of the memory usage of the key values corresponding to each node to the data as the slot memory usage of the slot according to the memory mirror file.
Similar to the manner of determining the number of slot requests, the server may also determine the key in each key pair, then determine the CRC code corresponding to the key according to the CRC algorithm, and determine the slot corresponding to the key according to the CRC code, thereby determining the slot memory usage of each slot.
And (III) the slot information is a resource consumption index of the corresponding node. The resource consumption indicators may include instruction response time consumption of the corresponding node, number of node requests, node memory usage, and node processor usage. The instruction response time is the average value of time spent by each node in response to the data query instruction, and the node processor utilization rate is the CPU (Central Processing Unit ) utilization rate of the node.
In this embodiment, a predetermined distributed system monitor may be deployed in advance in the server, and the instruction response time consumption, the number of node requests, the node memory usage amount, and the node processor usage rate of each node are obtained through the predetermined distributed system monitor.
The predetermined distributed system monitor of this embodiment may be zabbix. The Zabbix can monitor various network parameters and ensure the safe operation of the server system. Zabbix is made up of two parts, including a Zabbix server (Zabbix server) and a Zabbix agent (Zabbix agent), where the Zabbix server is deployed inside the server so that monitoring of each node can be provided by methods such as the Zabbix agent deployed to each node in a Redis cluster.
Step S200, determining an evaluation value of the corresponding slot according to the slot information.
In this step, the server may calculate a weighted sum of the slot information as an evaluation value of the corresponding slot point, respectively.
Alternatively, the server may acquire slot information of each slot, determine a weighted sum of the slot information according to a preset weight of each item of information, and use the calculated weighted sum as an evaluation value of the corresponding slot in the same period.
For example, the slot information of the slot S1 includes a slot request number of the slot S1, a slot memory usage amount, and a resource consumption index of a corresponding node, where the slot request number is 100, the memory usage amount is 0.2%, an instruction response time of the corresponding node in the resource consumption index is 3ms, the node request number is 1500, the node memory usage amount is 5%, and the node processor usage rate is 3%, and the server may obtain weights corresponding to each item of information in the above information, and calculate a weighted sum of the above information, so as to determine the weighted sum as an evaluation value of the slot S1.
Alternatively, the server may acquire slot information for each slot in a period including a predetermined number of periods, and then calculate a weighted sum of the same information in different periods as the slot information for the corresponding slot.
For example, the predetermined number is 5, and the number of slot requests of the slot S2 in different periods is 300,350, 480,630 and 1210, the server may obtain weights corresponding to the number of slot requests in each period, and calculate a weighted sum of the number of slot requests in different periods, thereby determining the weighted sum as the evaluation value of the slot S2.
Step S300, performing slot migration according to the evaluation value of each slot so as to balance the load of at least one slot.
In this step, the server may determine the slot to be migrated and the target slot according to the evaluation value of each slot in an iterative manner, and migrate the slot to be migrated to the target slot until a predetermined equalization condition is satisfied. The to-be-migrated slot may be a slot whose evaluation value satisfies a first condition, the target slot may be a slot whose evaluation value satisfies a second condition, and the predetermined balancing condition may be a slot request number of each slot, a slot memory usage amount, and instruction response time consumption of a corresponding node, a node request number, a difference value of at least part of information in the node memory usage amount and the node processor usage rate is smaller than a first threshold, or the like. The first condition may be set to evaluate the first n (n is a predetermined integer greater than or equal to 1) bits having a value less than (or equal to) the second threshold and ordered at the minimum, and the second condition may be set to evaluate the first m (m is a predetermined integer greater than or equal to 1) bits having a value greater than (or equal to) the third threshold and ordered at the maximum.
After determining the slot to be migrated and the target slot, the server may send cluster setslot importing instructions to the node where the target slot is located, change the state of the target slot to import, send cluster setslot migrating instructions to the node where the slot to be migrated is located, change the state of the slot to migrating (migration), then control the node where the slot to be migrated to execute cluster getkeysinslot instructions, obtain key value data in the slot to be migrated, and control the node where the slot to be migrated to migrate the obtained key value to the target node in batches through a pipeline (pipeline) mechanism until all the key value data in the slot to be migrated are migrated to the target node, so as to achieve the purpose of slot migration.
It is easy to understand that the method of the present embodiment may be performed at a predetermined period, that is, the server may perform steps S100 to S300 once in each period. Therefore, the server can load balance the slots in a preset period, so that the possibility that the node (namely the database) where one slot is located bears excessive access pressure is avoided.
Fig. 2 is a schematic diagram of a data processing method according to a first embodiment of the present invention. As shown in fig. 2, node (node) 1-node n is n nodes in the database cluster, and the ith (i is a predetermined integer of 1 or more and n or less) node includes mn slots, and zabbix agents (i.e., ZA) are pre-deployed inside each node. Therefore, the server can obtain the resource consumption index of each node through the internally deployed zabbix server, and obtain the slot request quantity of each slot and the slot memory usage quantity as the slot information of each slot. After the slot information of each slot is obtained, the server may calculate an evaluation value of each slot, determine that the slot to be migrated is the 1 st slot in the 1 st node, that is, slot1,1, and determine that the target slot is the nth slot in the nth node, that is, slot, mn. The server may then perform slot migration to balance the load of at least one slot.
In this embodiment, slot information of slots serving as a data storage range in a target database is determined, and evaluation values of the slots are determined according to the slot information of the slots, so that slot migration is performed according to the evaluation values of the slots, and load of at least one slot is balanced. The load balancing among the slots can balance the load of each node in the database cluster, so that the load of each database in the database cluster can be balanced, and the access pressure of a certain database can be effectively reduced.
Fig. 3 is a schematic diagram of a data processing apparatus according to a second embodiment of the present invention. As shown in fig. 3, the apparatus of the present embodiment includes an information determination unit 31, an evaluation value determination unit 32, and a slot migration unit 33.
The information determining unit 31 is configured to determine slot information of each slot in the target database cluster, where the slot is a data storage range. The evaluation value determination unit 32 is configured to determine an evaluation value of a corresponding slot according to each of the slot information. The slot shifting unit 33 is configured to perform slot shifting according to the evaluation value of each slot so as to balance the load of at least one slot.
In this embodiment, slot information of slots serving as a data storage range in a target database is determined, and evaluation values of the slots are determined according to the slot information of the slots, so that slot migration is performed according to the evaluation values of the slots, and load of at least one slot is balanced. The load balancing among the slots can balance the load of each node in the database cluster, so that the load of each database in the database cluster can be balanced, and the access pressure of a certain database can be effectively reduced.
Fig. 4 is a schematic view of an electronic device according to a third embodiment of the present invention. The electronic device shown in fig. 4 is a general-purpose data processing apparatus, and may specifically be a first terminal, a second terminal or a server according to an embodiment of the present invention, which includes a general-purpose computer hardware structure including at least a processor 41 and a memory 42. The processor 41 and the memory 42 are connected by a bus 43. The memory 42 is adapted to store instructions or programs executable by the processor 41. The processor 41 may be a separate microprocessor or may be a set of one or more microprocessors. Thus, the processor 41 performs the process of processing data and controlling other devices by executing the commands stored in the memory 42, thereby executing the method flow of the embodiment of the present invention as described above. The bus 43 connects the above components together, and connects the above components to the display controller 44 and the display device and the input/output (I/O) device 45. Input/output (I/O) device 45 may be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer, and other devices known in the art. Typically, an input/output (I/O) device 45 is connected to the system through an input/output (I/O) controller 46.
The memory 42 may store software components such as an operating system, communication modules, interaction modules, and application programs, among others. Each of the modules and applications described above corresponds to a set of executable program instructions that perform one or more functions and methods described in the embodiments of the invention.
The above-described flow diagrams and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention illustrate various aspects of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
Meanwhile, as will be appreciated by those skilled in the art, aspects of embodiments of the present invention may be implemented as a system, method, or computer program product. Accordingly, aspects of embodiments of the invention may take the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," module "or" system. Furthermore, aspects of the invention may take the form: a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.
Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of embodiments of the present invention, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, such as in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to: electromagnetic, optical, or any suitable combination thereof. The computer readable signal medium may be any of the following: a computer-readable storage medium is not a computer-readable storage medium and can communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including: object oriented programming languages such as Java, smalltalk, C ++, PHP, python, and the like; and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package; executing partly on the user computer and partly on the remote computer; or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, and various modifications and variations may be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A method of data processing, the method comprising:
determining slot position information of each slot position in a target database cluster, wherein the slot position is a data storage range;
determining an evaluation value of the corresponding slot according to the slot information;
performing slot migration according to the evaluation value of each slot so as to balance the load of at least one slot;
the slot information comprises the slot request quantity of each slot, the slot memory usage quantity and the resource consumption index of the corresponding node;
the determining the slot position information of each slot position in the target database cluster comprises the following steps:
acquiring a data query instruction received by an access layer;
determining a slot position corresponding to each data query instruction;
for each slot, taking the corresponding number of the data query instructions as the slot request number; or alternatively
Acquiring a memory mirror image file of the target database cluster;
analyzing the memory mirror image file and determining key value pair data in the memory mirror image file;
determining the corresponding slot position of each key value to the data;
for each slot, taking the sum of the memory usage of the corresponding key value pair data as the memory usage of the slot;
the performing slot migration according to the evaluation value of each slot includes:
and determining a to-be-migrated slot and a target slot according to each evaluation value in an iterative mode, and migrating the to-be-migrated slot to the target slot until a preset balance condition is met, wherein the to-be-migrated slot is a slot with the evaluation value meeting a first condition, and the target slot is a slot with the evaluation value meeting a second condition.
2. The method of claim 1, wherein the resource consumption indicators include instruction response time consumption, number of node requests, node memory usage, and node processor usage of the corresponding node;
the determining the slot position information of each slot position in the target database cluster further comprises:
and acquiring the instruction response time consumption, the node request quantity, the node memory usage and the node processor usage through a preset distributed system monitor.
3. The method of claim 1, wherein determining an evaluation value of a corresponding slot from each of the slot information comprises:
and respectively calculating the weighted sum of the slot position information as the evaluation value of the corresponding slot point.
4. A data processing apparatus, the apparatus comprising:
the information determining unit is used for determining slot position information of each slot position in the target database cluster, wherein the slot position is a data storage range;
an evaluation value determining unit for determining an evaluation value of a corresponding slot according to each slot information;
the slot position migration unit is used for carrying out slot position migration according to the evaluation value of each slot position so as to balance the load of at least one slot position;
the slot information comprises the slot request quantity of each slot, the slot memory usage quantity and the resource consumption index of the corresponding node;
the information determination unit is further configured to:
acquiring a data query instruction received by an access layer;
determining a slot position corresponding to each data query instruction;
for each slot, taking the corresponding number of the data query instructions as the slot request number; or alternatively
Acquiring a memory mirror image file of the target database cluster;
analyzing the memory mirror image file and determining key value pair data in the memory mirror image file;
determining the corresponding slot position of each key value to the data;
for each slot, taking the sum of the memory usage of the corresponding key value pair data as the memory usage of the slot;
the slot migration unit is further configured to:
and determining a to-be-migrated slot and a target slot according to each evaluation value in an iterative mode, and migrating the to-be-migrated slot to the target slot until a preset balance condition is met, wherein the to-be-migrated slot is a slot with the evaluation value meeting a first condition, and the target slot is a slot with the evaluation value meeting a second condition.
5. A computer readable storage medium, on which computer program instructions are stored, which computer program instructions, when executed by a processor, implement the method of any of claims 1-3.
6. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627012.9A CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011627012.9A CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112650737A CN112650737A (en) | 2021-04-13 |
| CN112650737B true CN112650737B (en) | 2024-03-19 |
Family
ID=75367347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011627012.9A Active CN112650737B (en) | 2020-12-31 | 2020-12-31 | Data processing method, data processing device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650737B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117439993B (en) * | 2023-12-22 | 2024-04-16 | 中电云计算技术有限公司 | Redis cluster load balancing method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2633413A2 (en) * | 2010-10-20 | 2013-09-04 | Microsoft Corporation | Low ram space, high-throughput persistent key-value store using secondary memory |
| CN105516367A (en) * | 2016-02-02 | 2016-04-20 | 北京百度网讯科技有限公司 | Distributed data storage system, method and device |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
-
2020
- 2020-12-31 CN CN202011627012.9A patent/CN112650737B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2633413A2 (en) * | 2010-10-20 | 2013-09-04 | Microsoft Corporation | Low ram space, high-throughput persistent key-value store using secondary memory |
| CN105516367A (en) * | 2016-02-02 | 2016-04-20 | 北京百度网讯科技有限公司 | Distributed data storage system, method and device |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
Non-Patent Citations (1)
| Title |
|---|
| 基于二次映射的哈希负载均衡方法;王永亮;;信息记录材料;20181201(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112650737A (en) | 2021-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110471821B (en) | Abnormality change detection method, server, and computer-readable storage medium | |
| CN109976975B (en) | Disk capacity prediction method and device, electronic equipment and storage medium | |
| CN110647490B (en) | Data transmission method, device, equipment and computer storage medium | |
| CN112231187B (en) | Micro-service abnormity analysis method and device | |
| US9910710B2 (en) | Prioritising event processing based on system workload | |
| US11341842B2 (en) | Metering data management system and computer readable recording medium | |
| CN109597798B (en) | Network file deleting method and device, computer equipment and storage medium | |
| US9384086B1 (en) | I/O operation-level error checking | |
| CN112650737B (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN111625592A (en) | Load balancing method and device for distributed database | |
| CN112860512A (en) | Interface monitoring optimization method and device, computer equipment and storage medium | |
| CN111562884B (en) | Data storage method and device and electronic equipment | |
| CN112380046A (en) | Calculation result checking method, system, device, equipment and storage medium | |
| CN110737548A (en) | Data request method and server | |
| CN116931829A (en) | Disk information transmission method, device, equipment and storage medium | |
| CN112671602A (en) | Data processing method, device, system, equipment and storage medium of edge node | |
| CN109829014B (en) | Method and device for acquiring test data | |
| CN109067649B (en) | Node processing method and device, storage medium and electronic equipment | |
| CN115129509B (en) | Data transmission method, device and medium | |
| CN116455697A (en) | Batch data recharging and checking method, system and terminal equipment | |
| CN113642638B (en) | Capacity adjustment method, model training method, device, equipment and storage medium | |
| CN113918237B (en) | Gray release method, device, equipment and computer readable medium | |
| CN115482869A (en) | Error injection testing method and device, electronic equipment and storage medium | |
| CN114513401A (en) | Automatic operation and maintenance repair method and device for private cloud and computer readable medium | |
| CN108512711B (en) | Method, device and equipment for obtaining instantaneous interface access statistic |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |