CN116414614A - Data processing method, storage medium and equipment for database cluster - Google Patents

Data processing method, storage medium and equipment for database cluster Download PDF

Info

Publication number
CN116414614A
CN116414614A CN202310421233.8A CN202310421233A CN116414614A CN 116414614 A CN116414614 A CN 116414614A CN 202310421233 A CN202310421233 A CN 202310421233A CN 116414614 A CN116414614 A CN 116414614A
Authority
CN
China
Prior art keywords
data
target data
processing method
result
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310421233.8A
Other languages
Chinese (zh)
Inventor
李翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingbase Information Technologies Co Ltd
Original Assignee
Beijing Kingbase Information Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingbase Information Technologies Co Ltd filed Critical Beijing Kingbase Information Technologies Co Ltd
Priority to CN202310421233.8A priority Critical patent/CN116414614A/en
Publication of CN116414614A publication Critical patent/CN116414614A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention provides a data processing method, a storage medium and equipment of a database cluster, wherein the data processing method of the database cluster comprises the following steps: the data sending process of the main database carries out first verification on the target data and generates a first verification result; the data sending process sends target data and a first verification result to a data receiving process of the backup database; the data receiving process of the standby database receives target data and a first verification result; the receiving data process performs second verification on the target data and generates a second verification result; judging whether the first checking result is consistent with the second checking result; if yes, writing the target data into a storage medium; if not, refusing to store the target data. The data processing method of the database cluster tries error correction as much as possible at the error node, and avoids storing error data due to network faults.

Description

Data processing method, storage medium and equipment for database cluster
Technical Field
The present invention relates to the field of databases, and in particular, to a data processing method, a storage medium, and a device for a database cluster.
Background
At present, most database products are based on a database cluster data redundancy technology, multiple data redundancy is carried out on one data per se, and when data errors occur in any cluster node data, in order to ensure data consistency, most of the existing products adopt a fault transfer and fault isolation mode to shield data nodes with data errors in clusters.
For example, the database master and slave clusters a, B, the data stream a sends B. When B receives the data information of A error or B writes the data information of A error after receiving the data of A, the database cluster AB data is inconsistent, at this time, the database cluster management system usually automatically deactivates the database B to keep the database A continuously providing service to the outside, and the business personnel select specific maintenance time to restore the database B.
Although the above-mentioned action well shields the mistake, but also makes the availability of the whole database cluster system reduce, when the above-mentioned AB is only A in work at last, it is in the single-point state, when there is A that there are other faults to happen, the database node B isolated can't participate in the fault state transition among them, cause the business system to stop using; meanwhile, due to the fact that the node B of the database cluster is in fault transfer or isolated, the corresponding added manual operation and maintenance cost is required to be monitored more frequently and manually participate in fault node recovery.
Disclosure of Invention
It is an object of the present invention to provide a data processing method, a storage medium and an apparatus of a database cluster capable of solving any of the above problems.
It is a further object of the invention to enhance the stability of a database cluster.
In particular, the present invention provides a data processing method of a database cluster, where the data processing method of the database cluster includes:
the data sending process of the main database carries out first verification on the target data and generates a first verification result;
the data sending process sends target data and a first verification result to a data receiving process of the backup database;
the data receiving process of the standby database receives target data and a first verification result;
the receiving data process performs second verification on the target data and generates a second verification result;
judging whether the first checking result is consistent with the second checking result;
if yes, writing the target data into a storage medium;
if not, refusing to store the target data.
Further, the step of writing the target data to the storage medium further includes:
the playback data process attempts to read the target data;
and judging whether the target data can be read normally according to the trial result.
Further, the step of judging whether the target data can be read normally according to the trial result includes:
and under the condition that the result of the attempt is successful, determining that the target data can be read normally.
Further, the step of judging whether the target data can be read normally according to the result of the attempt further comprises:
under the condition that the trial result is failure, judging that the target data can not be read normally;
the playback data process notifies the receiving data process that waits for the next received data.
Further, the step of waiting for the next received data by the data receiving process includes:
the receiving data process returns to the starting position of the receiving target data, and the sending data process resends the target data.
Further, in the case that the first check result is inconsistent with the second check result, the method further includes:
the send data process resends the target data.
Further, in the case that the first check result is inconsistent with the second check result, the method further includes:
and returning the received data process to the starting position of the received target data.
Further, the check is a cyclic redundancy check.
According to another aspect of the present invention, there is also provided a machine-readable storage medium having stored thereon a machine-executable program which, when executed by a processor, implements a data processing method of any one of the above database clusters.
According to another aspect of the present invention, there is also provided a computer device including a memory, a processor, and a machine executable program stored on the memory and running on the processor, and the processor implementing a data processing method of any one of the above database clusters when executing the machine executable program.
The data processing method of the database cluster comprises the following steps: the data sending process of the main database carries out first verification on the target data and generates a first verification result; the data sending process sends the target data and the first verification result to a data receiving process of the backup database; the data receiving process of the standby database receives the target data and the first verification result; the receiving data process performs second verification on the target data and generates a second verification result; judging whether the first checking result is consistent with the second checking result; if yes, writing the target data into a storage medium; if not, refusing to store the target data. The data processing method of the database cluster tries error correction as much as possible at the error node, and avoids storing error data due to network faults.
Further, the data processing method of the database cluster of the present invention further includes, after the step of writing the target data to the storage medium: a playback data process attempts to read the target data; judging whether the target data can be read normally or not according to the try result. The step of judging whether the target data can be read normally according to the trial result further comprises the following steps: under the condition that the trial result is failure, judging that the target data can not be read normally; the playback data process notifies the receiving data process, which waits for the next received data. The data processing method of the database cluster utilizes the data retransmission to try error correction at the error node as much as possible, improves the consistency of the cluster data and ensures that the cluster system is recovered to be normal.
The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.
Drawings
Some specific embodiments of the invention will be described in detail hereinafter by way of example and not by way of limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions. It will be appreciated by those skilled in the art that the drawings are not necessarily drawn to scale. In the accompanying drawings:
FIG. 1 is a schematic diagram of a data flow of a database cluster according to one embodiment of the invention;
FIG. 2 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention;
FIG. 3 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention;
FIG. 4 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention
FIG. 5 is a schematic diagram of a machine-readable storage medium according to one embodiment of the invention; and
FIG. 6 is a schematic diagram of a computer device according to one embodiment of the invention.
Detailed Description
FIG. 1 is a schematic diagram of a data flow of a database cluster according to one embodiment of the invention. As shown in fig. 1, the database cluster of the present embodiment may generally include a primary database a and a backup database B. The data flow is sent to the standby database B for the primary database A. The process A (walsender A) of the primary database a is a sending data process a (sender a), and the process B (walsender B) of the backup database B is a receiving data process B (receiver B). When the primary database a attempts to transmit the target data data_1 to the backup database B, the CRC check of the target data data_1 is automatically calculated by the transmitting data process a, a check result crc_data_1 is generated, and data_1 and crc_data_1 are concurrently transmitted to the backup database B, as indicated by an arrow pointing to the primary database a by data_1 in fig. 1. When the receiving data process B of the standby database B receives the data_1 and the crc_data_1, it automatically calculates whether the crc check value of the data_1 and the crc_data_1 are equal, as indicated by the arrow pointing from the main database a to the standby database B in fig. 1. In the case where the crc check value of data_1 calculated by the reception data process B is equal to crc_data_1, the reception data process B returns to the data_1 reception end position, and the spare database B writes data_1 to the storage medium, as indicated by the arrow pointing to data_1 by the spare database B in fig. 1. The reading of data is performed by the playback data process B (redox B) of the backup database B and applied to the backup database B, as indicated by the arrow pointing to the playback data process B by data_1 in fig. 1. In the case where playback data process B reads data_1 and playback is impossible, playback data process B notifies reception data process B, as indicated by an arrow pointing to reception data process B by playback data process B in fig. 1. The receiving data process B returns to the starting position of the receiving data_1 and notifies the sending data process A to wait for the sending data process A to send the data_1 again.
FIG. 2 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention. As shown in fig. 2, the data processing method of the database cluster of the present embodiment includes:
step S202, the primary data transmission process performs primary verification on target data, and generates a primary verification result crc_data_1. The check may be a cyclic redundancy check (CRC, cyclic redundancy check). The cyclic redundancy check is a hash function for generating a short fixed bit check code according to data such as computer files, and is mainly used for detecting or checking errors possibly occurring after data transmission or storage. The generated numbers are calculated before transmission or storage and appended to the data, and then the recipient checks to determine if the data has changed.
Step S204, the sending data process sends the target data and the first check result to the receiving data process of the backup database.
In step S206, the data receiving process of the standby database receives the target data and the first verification result.
Step S208, the receiving data process performs a second verification on the target data, and generates a second verification result.
Step S210, judging whether the first checking result is consistent with the second checking result. If yes, go to step S212; if not, step S214 is performed.
Step S212, the target data is written to the storage medium.
Step S214, refusing to store the target data, retransmitting the target data by the data transmission process, and returning the data reception process to the initial position of receiving the target data.
In the data processing method of the database cluster in this embodiment, when the database data is transmitted, the main database automatically calculates the CRC value of the transmitted target data and transmits the CRC value to the standby database, and the standby database also automatically checks the data when receiving the data and compares whether the transmitted CRC value is consistent. And the data is stored only when the data are consistent, so that the error data are prevented from being stored. And when the data is inconsistent, the data sending end is informed to resend the data, so that the packet receiving abnormality caused by network faults is effectively avoided, and the reliability of the data transmission of the database cluster system is improved.
FIG. 3 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention. As shown in fig. 3, the method for processing data in a database cluster according to the present embodiment further includes, after the step of writing the target data into a storage medium:
in step S302, the playback data process attempts to read the target data.
Step S304, judging whether the target data can be read normally according to the trial result.
The data processing method of the database cluster in the embodiment determines whether the situation of the data error occurs or not through playing back the target data, and can be used for subsequent timely correction.
Fig. 4 is a schematic step diagram of a data processing method of a database cluster according to one embodiment of the invention. As shown in fig. 4, in the data processing method of the database cluster of the present embodiment, the step of determining whether the target data can be read normally according to the result of the attempt includes:
in step S402, the playback data process attempts to read the target data. If the result of the attempt is successful, step S404 is performed; in the case where the result of the attempt is failure, step S406 is performed.
In step S404, it is determined that the target data can be read normally, and no other operations are performed.
In step S406, it is determined that the target data is not normally readable.
In step S408, the playback data process notifies the receiving data process, and the receiving data process waits for the next receiving data. Wherein the step of the data receiving process waiting for the next data receiving process may include: and returning the starting position for receiving the target data by the data receiving process, and retransmitting the target data by the data transmitting process.
After the target data of the database is written into the storage medium, if unexpected values of the read data occur, the data processing method of the database cluster automatically corrects the initial position of the target data to be obtained again by the receiving data process and feeds the initial position back to the sending data process, and at the moment, the sending data process can resend all data from the data request initial position Jin Cining palace, thereby effectively avoiding the read data abnormality of the standby database due to the storage medium or other factors and further improving the transmission data reliability of the database cluster system.
The data processing method of the database cluster of the embodiment utilizes the fault detection and the full data of the 2 stages, sacrifices a little performance, only provides read-only inquiry for ensuring the data consistency during the automatic recovery period (the time T1 is usually in the millisecond-second level), and returns the application service to normal after the recovery window is finished.
The data processing method of the database cluster removes the fault transfer and fault isolation in the conventional scheme, so that the faulty node resources are not seriously wasted, and the manual intervention is waited. Instead, the error node tries error correction as much as possible, and the cluster data becomes consistent again by using data retransmission. In such a fault scene of common data mistransmission and data writing errors in database data transmission, the reliability of the whole cluster system is greatly improved, and the cluster system can be ensured to recover normally only by tolerating an acceptable T1 time period when encountering such faults; the cluster management cost is reduced, and the manual intervention recovery cluster cost is also reduced.
The present embodiment also provides a machine-readable storage medium and a computer device. Fig. 5 is a schematic diagram of a machine-readable storage medium according to one embodiment of the invention, and fig. 6 is a schematic diagram of a computer device according to one embodiment of the invention.
The machine-readable storage medium 40 has stored thereon a machine executable program 41, which when executed by a processor, implements the data processing method of the database cluster of any of the embodiments described above.
The computer device 50 may include a memory 520, a processor 510, and a machine executable program 41 stored on the memory 520 and running on the processor 510, and the processor 510 implements the data processing method of the database cluster of any of the embodiments described above when executing the machine executable program 41.
It should be noted that the logic and/or steps represented in the flow diagrams or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any machine-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
For the purposes of this description of embodiments, a machine-readable storage medium 40 can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium 40 may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system.
The computer device 50 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, computer device 50 may be a cloud computing node. Computer device 50 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer device 50 may be implemented in a distributed cloud computing environment where remote processing devices coupled via a communications network perform tasks. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.
Computer device 50 may include a processor 510 adapted to execute stored instructions, a memory 520 providing temporary storage for the operation of the instructions during operation. Processor 510 may be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 520 may include Random Access Memory (RAM), read only memory, flash memory, or any other suitable storage system.
The processor 510 may also be linked through a system interconnect to a display interface suitable for connecting the computer device 50 to a display device. The display device may include a display screen as a built-in component of the computer device 50. The display device may also include a computer monitor, television, projector, or the like, that is externally connected to the computer device 50. Further, a network interface controller (network interface controller, NIC) may be adapted to connect the computer device 50 to a network through a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an internet small computer system interface, etc.) to transfer data. The network may be a cellular network, a radio network, a Wide Area Network (WAN), a Local Area Network (LAN), or the internet, among others. The remote device may be connected to the computing device through a network.
The flowcharts provided by this embodiment are not intended to indicate that the operations of the method are to be performed in any particular order, or that all of the operations of the method are included in all of each case. Furthermore, the method may include additional operations. Additional variations may be made to the above-described methods within the scope of the technical ideas provided by the methods of the present embodiments.
By now it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been shown and described herein in detail, many other variations or modifications of the invention consistent with the principles of the invention may be directly ascertained or inferred from the present disclosure without departing from the spirit and scope of the invention. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Claims (10)

1. A data processing method of a database cluster, comprising:
the data sending process of the main database carries out first verification on the target data and generates a first verification result;
the data sending process sends the target data and the first verification result to a data receiving process of the backup database;
the data receiving process of the standby database receives the target data and the first verification result;
the receiving data process performs second verification on the target data and generates a second verification result;
judging whether the first checking result is consistent with the second checking result;
if yes, writing the target data into a storage medium;
if not, refusing to store the target data.
2. The data processing method of a database cluster according to claim 1, wherein the step of writing the target data to a storage medium further comprises:
a playback data process attempts to read the target data;
judging whether the target data can be read normally or not according to the try result.
3. The data processing method of a database cluster according to claim 2, wherein the step of judging whether the target data is normally readable according to the trial result comprises:
and if the result of the attempt is successful, determining that the target data can be read normally.
4. A data processing method of a database cluster according to claim 3, wherein the step of judging whether the target data is normally readable according to the result of the attempt further comprises:
under the condition that the trial result is failure, judging that the target data can not be read normally;
the playback data process notifies the receiving data process, which waits for the next received data.
5. The data processing method of a database cluster according to claim 4, wherein the step of receiving the data process waiting for the next received data comprises:
and the receiving data process returns to the starting position for receiving the target data, and the sending data process resends the target data.
6. The data processing method of a database cluster according to claim 1, wherein, in the case that the first check result is inconsistent with the second check result, further comprising:
the sending data process resends the target data.
7. The data processing method of a database cluster according to claim 1, wherein, in the case that the first check result is inconsistent with the second check result, further comprising:
and returning the received data process to the starting position for receiving the target data.
8. The method for processing data of a database cluster according to claim 1, wherein,
the check is a cyclic redundancy check.
9. A machine readable storage medium having stored thereon a machine executable program which when executed by a processor implements a data processing method of a database cluster according to any of claims 1 to 8.
10. A computer device comprising a memory, a processor and a machine executable program stored on the memory and running on the processor, and the processor implementing a data processing method of a database cluster according to any one of claims 1 to 8 when executing the machine executable program.
CN202310421233.8A 2023-04-18 2023-04-18 Data processing method, storage medium and equipment for database cluster Pending CN116414614A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310421233.8A CN116414614A (en) 2023-04-18 2023-04-18 Data processing method, storage medium and equipment for database cluster

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310421233.8A CN116414614A (en) 2023-04-18 2023-04-18 Data processing method, storage medium and equipment for database cluster

Publications (1)

Publication Number Publication Date
CN116414614A true CN116414614A (en) 2023-07-11

Family

ID=87052856

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310421233.8A Pending CN116414614A (en) 2023-04-18 2023-04-18 Data processing method, storage medium and equipment for database cluster

Country Status (1)

Country Link
CN (1) CN116414614A (en)

Similar Documents

Publication Publication Date Title
CN110349029B (en) Block chain-based transaction consistency processing method, device and system
WO1991014230A1 (en) Message communication processing system
EP2432155B1 (en) Call list transmission processing method and system of charging system
WO2020232859A1 (en) Distributed storage system, data writing method, device, and storage medium
CN112130881A (en) Firmware remote upgrading method and device
CN111478796B (en) Cluster capacity expansion exception handling method for AI platform
US20090138757A1 (en) Failure recovery method in cluster system
CN113986501A (en) Real-time database API (application program interface) uninterrupted calling method, system, storage medium and server
CN112153132A (en) File uploading method, device and equipment based on virtualization management platform
CN112395237A (en) Method and system for communication between at least two controllers
JP2007013980A (en) Redundant three-wire communication system and method
CN106598690B (en) Management method and device for codes
CN116414614A (en) Data processing method, storage medium and equipment for database cluster
US20030154288A1 (en) Server-client system and data transfer method used in the same system
CN108599903B (en) Single board starting control method and device
CN113452603A (en) Data transmission method and related device
CN107346273B (en) Data recovery method and device and electronic equipment
CN114362893A (en) Data transmitting method, data receiving method, terminal and computer storage medium
JP4710719B2 (en) Retransmission device when communication is abnormal
CN114745281B (en) Data processing method and device
CN112311502B (en) Data retransmission method, device, controller and medium based on optical fiber communication system
CN114124790B (en) Data transmission method and device based on directed acyclic graph and electronic equipment
CN116361282A (en) Data processing method, device, related equipment and storage medium
CN115633046A (en) Kafka high-availability scheme optimization method, device, equipment and storage medium
CN116962387A (en) Data transmission method and system for Ethernet communication

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination