CN116186030A - Data processing method and device of database and computer equipment - Google Patents
Data processing method and device of database and computer equipment Download PDFInfo
- Publication number
- CN116186030A CN116186030A CN202211645517.7A CN202211645517A CN116186030A CN 116186030 A CN116186030 A CN 116186030A CN 202211645517 A CN202211645517 A CN 202211645517A CN 116186030 A CN116186030 A CN 116186030A
- Authority
- CN
- China
- Prior art keywords
- task
- determining
- data
- target task
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data processing method, a device and computer equipment of a database, wherein the method comprises the following steps: determining a target task according to the task identification, wherein the target task is a task responding to overtime; determining task execution information of the target task; and generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
Description
Technical Field
The disclosure relates to the technical field of databases, and in particular relates to a data processing method and device of a database and computer equipment.
Background
In a distributed database system, a plurality of data storage units are typically included, which are connected to form a logically unified database. When a transaction is initiated in the distributed database, data operations may be performed based on the data storage units involved in the transaction.
However, due to different states of the data storage units, the execution of the transaction in a part of the data storage units may fail, and at this time, the transaction often needs to be rolled back, specifically, a rollback statement of the transaction may be generated by analyzing an operation log corresponding to the transaction, so as to re-execute the transaction based on the rollback statement. However, in the existing technical solution, a large amount of operation logs are often required to be parsed, which takes a long time, so that a large amount of data is in an unavailable state for a long time, and meanwhile, a condition that log files are cleaned may exist, so that log files required by rollback cannot be found, and rollback failure is caused.
Disclosure of Invention
The embodiment of the disclosure at least provides a data processing method, a data processing device and computer equipment for a database.
In a first aspect, an embodiment of the present disclosure provides a data processing method of a database, including:
determining a target task according to the task identification, wherein the target task is a task responding to overtime;
determining task execution information of the target task;
and generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
In an alternative embodiment, the determining the target task according to the task identifier includes:
determining task execution time corresponding to the task identifier;
and under the condition that the task execution time exceeds a time threshold, determining the task corresponding to the task identifier as a target task.
In an optional embodiment, the determining the task execution time corresponding to the task identifier includes:
determining the creation time of the task identifier;
and under the condition that the preset operation of the task identifier is not received, determining the task execution time corresponding to the task identifier based on the current moment and the creation time, wherein the preset operation is used for indicating ending the operation of the task corresponding to the task identifier.
In an alternative embodiment, the method further comprises:
in the case of receiving a configuration request, determining a time threshold indicated by the configuration request and a polling time;
and determining a monitoring operation for monitoring the task execution time based on the polling time, wherein the monitoring operation is used for indicating the time length of comparing the task execution time with a time threshold.
In an alternative embodiment, the method further comprises:
after determining a target task according to a task identifier, determining task information of the target task, wherein the task information comprises: a data storage location identifier and an execution object identifier;
determining an analysis object corresponding to the target task based on the task information; the analysis object is used for analyzing the task execution information and generating rollback data of the target task.
In an optional implementation manner, the determining, based on the task information, the analysis object corresponding to the target task includes:
determining a data storage position corresponding to the target task based on the data storage position identifier;
and determining an execution object matched with the data storage position, and determining an analysis object corresponding to the data storage position based on the execution object.
In an alternative embodiment, the determining the task execution information of the target task includes:
determining index information corresponding to the target task;
and searching task execution information corresponding to the task identifier in a task log based on the index information.
In an alternative embodiment, the method further comprises:
after the rollback data of the target task is generated, determining the storage state of the rollback data;
based on the storage state, under the condition that the rollback data is determined to be failed to be stored, establishing a storage period of the rollback data;
and executing storage operation on the rollback data according to the storage period.
In an alternative embodiment, the performing the target task based on the rollback data includes:
acquiring local data of the target task, and inquiring rollback data of the target task based on the local data;
determining operation sentences and task data of the target task based on the rollback data;
and executing the task data based on the operation statement.
In a second aspect, an embodiment of the present disclosure further provides a data processing apparatus of a database, including:
The first determining unit is used for determining a target task according to the task identification, wherein the target task is a task responding to overtime;
the second determining unit is used for determining task execution information of the target task;
and the generating unit is used for generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
In a third aspect, embodiments of the present disclosure further provide a computer device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the computer device is running, the machine-readable instructions when executed by the processor performing the steps of the first aspect, or any of the possible implementations of the first aspect.
In a fourth aspect, the presently disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect, or any of the possible implementations of the first aspect.
In the embodiment of the disclosure, firstly, a target task with response overtime can be determined according to the task identifier, so that monitoring of the task with response overtime is realized, and after the target task is determined, the task execution information of the target task can be determined, so that the data volume of the task execution information required to be analyzed is reduced. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
The foregoing objects, features and advantages of the disclosure will be more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.
FIG. 1 illustrates a flow chart of a method for processing data of a database provided by an embodiment of the present disclosure;
FIG. 2 illustrates a flow chart of another database data processing method provided by an embodiment of the present disclosure;
FIG. 3 illustrates a schematic frame structure of a data processing system of a database provided by an embodiment of the present disclosure;
FIG. 4 illustrates a schematic diagram of the operation of a management module provided by an embodiment of the present disclosure;
FIG. 5 shows a schematic diagram of a data processing apparatus of a database provided by an embodiment of the present disclosure;
Fig. 6 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. The components of the embodiments of the present disclosure, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be made by those skilled in the art based on the embodiments of this disclosure without making any inventive effort, are intended to be within the scope of this disclosure.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The term "and/or" is used herein to describe only one relationship, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
It has been found that a distributed database system typically includes a plurality of data storage units that are connected to form a logically unified database. When a transaction is initiated in the distributed database, data operations may be performed based on the data storage units involved in the transaction.
However, due to different states of the data storage units, the execution of the transaction in a part of the data storage units may fail, and at this time, the transaction often needs to be rolled back, specifically, a rollback statement of the transaction may be generated by analyzing an operation log corresponding to the transaction, so as to re-execute the transaction based on the rollback statement. However, in the existing technical solution, a large amount of operation logs are often required to be parsed, which takes a long time, so that a large amount of data is in an unavailable state for a long time, and meanwhile, a condition that log files are cleaned may exist, so that log files required by rollback cannot be found, and rollback failure is caused.
Based on the above study, the present disclosure provides a data processing method and apparatus for a database, and a computer device. In the embodiment of the disclosure, firstly, a target task with response overtime can be determined according to the task identifier, so that monitoring of the task with response overtime is realized, and after the target task is determined, the task execution information of the target task can be determined, so that the data volume of the task execution information required to be analyzed is reduced. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
For the sake of understanding the present embodiment, first, a detailed description will be given of a data processing method of a database disclosed in the embodiments of the present disclosure, where an execution body of the data processing method of the database provided in the embodiments of the present disclosure is generally a computer device with a certain computing capability. In some possible implementations, the data processing method of the database may be implemented by a processor calling computer readable instructions stored in a memory.
Referring to fig. 1, a flowchart of a method for processing data in a database according to an embodiment of the disclosure is shown, where the method includes steps S101 to S105, where:
s101: and determining a target task according to the task identification, wherein the target task is a task responding to the overtime.
In the embodiment of the present disclosure, when executing a task in the above-mentioned distributed database system, a corresponding task identifier may be first generated for the task, and whether the task is executed is determined based on the task identifier, where the task identifier may be denoted as gtid. Specifically, the gtid of the task can be generated through the gtid management module in the distributed database system, when the task is data storage in the data storage unit in the distributed database system, the gtid of the task can be stored in the data storage unit at the same time, and after the task is executed, a task ending instruction can be fed back to the gtid management module, so that the gtid management module is instructed to release the gtid of the task.
Based on the above, whether the task identifier is released by the gtid management module can be detected in real time, and when the duration that the task identifier is not released by the gtid management module exceeds a time threshold, a task response timeout corresponding to the task identifier can be determined, and the task is determined to be a target task, wherein the task can be a task which fails to be executed in at least part of the data storage units.
S103: and determining task execution information of the target task.
In the embodiment of the present disclosure, the task execution information may include an operation log corresponding to the target task, and when determining that the task execution information is red, log data may be obtained in real time when detecting that the target task is a task with response time-out, and the operation log corresponding to the target task is determined in the log data, where the operation log may include an operation sentence, a sentence type, task data, and the like corresponding to the target task.
S105: and generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
In the embodiment of the present disclosure, the rollback data may include rsql (rollback sql, database rollback statement), where the rsql may be used to indicate contents such as an operation statement, a statement type, etc. corresponding to a target task, and task data corresponding to the target task may be processed based on the rsql to execute the target task.
After the rollback data is determined, the rollback data can be stored in a user directory corresponding to the target task, so that pre-rollback for the target task is completed. When a rollback request for the target task is received, the user catalog can be searched first, rollback data corresponding to the target task can be directly obtained after the rollback data is searched, and the target task is executed based on the rollback data, so that task execution information of the target task is not required to be obtained again, rollback is performed based on the task execution information to generate the rollback data, rollback time of the target task is effectively reduced, and the rollback time reducing effect is more obvious through the pre-rollback mode of the present disclosure under the condition that the task quantity of the target task is larger.
As can be seen from the foregoing description, in the embodiment of the present disclosure, a target task with a response timeout may be determined according to a task identifier, so as to monitor the task with the response timeout, and after the target task is determined, task execution information of the target task may be determined, so as to reduce the data size of the task execution information that needs to be parsed. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
In an optional embodiment, the step S101 determines the target task according to the task identifier, which specifically includes the following steps:
S1011: and determining the task execution time corresponding to the task identifier.
S1012: and under the condition that the task execution time exceeds a time threshold, determining the task corresponding to the task identifier as a target task.
In the embodiment of the disclosure, a user can start a timing detection task through the operation and maintenance management platform in the distributed database system, and after detecting that the timing detection task is started, the user can monitor a task identifier in the gtid management module to determine the task execution time of a corresponding task based on the task identifier. Here, the task execution time may be used to indicate a time when the task identification of the target task is not released.
Next, the time threshold configured by the above-described timing detection task may be acquired, and whether the task execution time exceeds the time threshold may be detected in real time, and in the case where the task execution time exceeds the time threshold, it may be determined that the task is likely to fail to be executed in at least part of the data storage units, and the task may be determined as the target task.
In the embodiment of the disclosure, the task execution time of the task corresponding to the task identification can be monitored in real time by detecting the task at regular time, so that the efficiency of determining the target task is improved, the data volume of the task execution information required to be analyzed in the next rollback process is reduced, and the time consumption for analyzing the task execution information is reduced.
In an optional embodiment, the step S1011, determining the task execution time corresponding to the task identifier specifically includes the following steps:
(1) Determining the creation time of the task identifier;
(2) And under the condition that the preset operation of the task identifier is not received, determining the task execution time corresponding to the task identifier based on the current moment and the creation time, wherein the preset operation is used for indicating the operation of ending the task corresponding to the task identifier.
In the embodiment of the disclosure, after a task identifier is created for a task, the creation time of the task identifier may be recorded, and then a preset operation for the task identifier may be detected, where the preset operation may be the release operation of the gtid management module for the task identifier. Next, the task execution time of the task corresponding to the task identifier may be monitored in real time, and specifically, the task execution time of the task may be determined based on a difference between the current time and the creation time.
Based on this, in the embodiment of the present disclosure, the task execution time of the task identifier corresponding to the task may be monitored in real time, so that when the task execution time exceeds the time threshold, the task is determined to be the target task, so that the data amount of the task execution information required to be traced back when the rollback is performed for the target task is reduced, so as to reduce the time consumption for analyzing the task execution information.
In an alternative embodiment, the embodiment corresponding to step S101 further includes the following procedure:
s11: in the case of receiving a configuration request, a time threshold indicated by the configuration request and a polling time are determined.
S12: and determining a monitoring operation for monitoring the task execution time based on the polling time, wherein the monitoring operation is used for indicating the time length of comparing the task execution time with a time threshold.
In an embodiment of the present disclosure, the detection parameters of the timing detection task may be configured by the operation and maintenance management platform, and specifically, the detection parameters may include a time threshold, a polling time, and a tenant id, where the polling time may be used to indicate a detection period when a task execution time of the detection task exceeds the time threshold, the tenant id may be used to indicate a detection range of the timing detection task, and each tenant may correspond to at least one data storage unit in the distributed database system.
After the detection parameters are determined, the detection parameters can be configured into a management module through an operation and maintenance management platform so as to execute the timing detection task through the management module, and in particular, when the management module executes monitoring operation based on the timing detection task, the management module can periodically detect whether the task execution time of the task matched with the detection range exceeds a time threshold according to the polling time.
It should be understood that when the detection parameters are configured in the management module, a configuration mode with dynamic effect can be adopted, and at this time, the management module can directly use the detection parameters to perform the monitoring operation without restarting the timing detection task, thereby improving the execution efficiency of the monitoring operation.
Meanwhile, in the embodiment of the disclosure, whether the configuration of the detection parameter is successful or not can be monitored, if the configuration is successful, a configuration success signal is fed back to the operation and maintenance management platform, if the configuration is failed, an error may exist in the detection parameter, at this time, the error may be returned to the operation and maintenance management platform, so that the operation and maintenance management platform modifies the detection parameter, and the modified detection parameter is reconfigured to the management module, thereby completing self-checking and error correction for the detection parameter.
In the embodiment of the disclosure, the detection parameters of the timing detection task can be flexibly configured, so that the corresponding detection parameters can be configured according to different use situations, and the practicability of the disclosure is improved.
In an alternative embodiment, the foregoing embodiment corresponding to fig. 1 further includes step S102, where step S102 specifically includes the following procedures:
S1021: after determining a target task according to a task identifier, determining task information of the target task, wherein the task information comprises: data storage location identification, execution object identification.
S1022: determining an analysis object corresponding to the target task based on the task information; the analysis object is used for analyzing the task execution information and generating rollback data of the target task.
In the embodiment of the present disclosure, after the gtid management module determines the target task, task information of the target task may be obtained, where the task information may include: data storage location identification, execution object identification, and task identification. The data storage location identifier may be the tenant id, where the tenant id is used to indicate an identifier of a data storage unit corresponding to the target task, and the execution object identifier may be used to indicate an execution object of the target task, where the execution object may be a server allocated for the tenant id. For example, when the target task is to write data into a data storage unit, the execution object may perform a write operation to write the data into the corresponding data storage unit.
After determining the task information of the target task, analysis may be performed based on the task information, so as to determine an analysis object corresponding to the target task, so as to analyze the task execution information of the target task based on the analysis object, so as to determine rollback data of the target task, and a specific process of determining the analysis object is described below, which is not described herein.
In the embodiment of the disclosure, the analysis object corresponding to the target task can be determined based on the task information of the target task, so that the task execution information corresponding to the target task is analyzed based on the analysis object, thereby improving the analysis rate and further improving the pre-rollback efficiency.
In an optional embodiment, step S1022 above, based on the task information, determines an analysis object corresponding to the target task, and specifically further includes the following steps:
(1) Determining a data storage position corresponding to the target task based on the data storage position identification;
(2) And determining an execution object matched with the data storage position, and determining an analysis object corresponding to the data storage position based on the execution object.
In the embodiment of the disclosure, firstly, a data storage position corresponding to the target task can be determined based on a data storage position identification in task information, wherein the data storage position can be used for indicating a corresponding data storage unit of the target task in a distributed database system.
It should be understood that each data storage unit in the distributed database system may be pre-allocated with a corresponding execution object, where the execution object may be a computing module such as a server, and may perform an operation on a task related to the data storage unit through the execution object.
In addition, when the number of execution objects allocated to the data storage unit is plural, the management module may determine that another execution object allocated to the data storage unit is an execution object matching the data storage unit when the determined execution object is in an abnormal state, or determine that the execution object is an execution object matching the data storage unit after the abnormal state of the execution object is recovered.
After determining an execution object matched with a data storage unit corresponding to a target task, determining an analysis object corresponding to the data storage unit based on the execution object, wherein the analysis unit may be an analysis module capable of analyzing task execution information. Specifically, a corresponding parsing module may be pre-established for each data storage unit in the distributed database system, for example, at least one parsing module may be set for one data storage unit, for example, when a plurality of data partitions are included in the data storage unit, a corresponding parsing module may be set for each data partition, so as to parse task execution information corresponding to the data partition based on the parsing module.
In the embodiment of the disclosure, a corresponding analysis object may be set in advance for the data storage unit, so as to analyze task execution information corresponding to the target task based on the analysis object, thereby improving analysis rate and further improving pre-rollback efficiency.
In an optional embodiment, the step S105 determines task execution information of the target task, which specifically includes the following steps:
s1051: and determining index information corresponding to the target task.
S1052: and searching task execution information corresponding to the task identifier in a task log based on the index information.
In the embodiment of the present disclosure, it is known from the foregoing that the task execution information may include an operation log corresponding to the target task, and the operation log may be generated for the operation behavior in the distributed database system, for example, the operation log may be generated for the operation of executing the task.
Therefore, the amount of data included in the operation log is often huge, and the time consumed for searching the operation log corresponding to the target task is also long.
In the process of determining the task execution information of the target task, a task log related to the execution task can be determined in the operation log of the distributed database system, the operation log with a mapping relation with the task identifier of the target task in the task log is searched based on the index information, and the operation log is determined to be the task execution information of the target task.
In the embodiment of the disclosure, index information may be preset to determine an operation log corresponding to a task identifier of a target task based on the index information, where the index information may include a mapping relationship between the task identifier of the target task and the corresponding operation log, so as to improve efficiency of determining the operation log of the target task.
In an alternative embodiment, the foregoing embodiment corresponding to fig. 1 further includes the following procedure:
s21: after the rollback data of the target task is generated, the storage state of the rollback data is determined.
S22: and establishing a storage period of the rollback data under the condition that the rollback data is determined to be failed to be stored based on the storage state.
S23: and executing storage operation on the rollback data according to the storage period.
In the embodiment of the present disclosure, it is known from the above that the rollback data may be stored in a user directory corresponding to a target task, so as to complete pre-rollback for the target task. However, since the states of the storage areas in the distributed database system may not be consistent, the user directory corresponding to the target task may be in a failure state and cannot store the rollback data, for example, the data storage unit corresponding to the user directory is in a power-off state.
Based on the above, after the rollback data of the target task is generated by the pre-rollback method in the disclosure, the storage state of the rollback data can be monitored, and after the storage error is received, the storage state of the rollback data can be determined to be storage failure.
In an alternative embodiment, the storage data may be stored in a buffer space, and a storage period of the rollback data may be determined, and then the rollback data may be periodically acquired from the buffer space according to the storage period, and a storage operation may be performed on the rollback data.
In another optional embodiment, if it is determined that the rollback data storage fails, if rollback data in the cache space is not acquired, the management module may be notified to re-execute a process of determining rollback data of the target task, where a specific manner of determining rollback data in the process is described above, and will not be described herein.
In the embodiment of the disclosure, since the states of the storage areas in the distributed database system may be inconsistent, the user directory corresponding to the target task may be in a fault state and cannot store the rollback data, for example, the data storage unit corresponding to the user directory is in a power-off state. Based on the above, according to the above storage period, a periodic storage operation can be performed on the rollback data, so as to improve the success rate of storing the rollback data, thereby perfecting the flow of the pre-rollback mode in the disclosure.
In an optional embodiment, step S105 above performs the target task based on the rollback data, and specifically includes the following procedures:
s1051: and acquiring the local data of the target task, and inquiring the rollback data of the target task based on the local data.
S1052: and determining the operation statement and task data of the target task based on the rollback data.
S1053: and executing the task data based on the operation statement.
In the embodiment of the disclosure, an execution instruction for a target task may be detected first, and whether rollback data of the target task exists in local data corresponding to the target task is queried in response to the execution instruction.
Then, the rollback data may be parsed to obtain rsql of the target task, that is, an operation statement, and task data corresponding to the target task, and the task data may be executed based on the operation statement, so as to re-execute the target task.
Additionally, in the embodiment of the present disclosure, a local cleaning period may also be configured to periodically clean the rollback data in the local data, so as to avoid overflow of the locally stored content. Specifically, after the cleaning period is reached, the executed rollback data may be identified and cleared, or after the cleaning period is reached, all rollback data in the local data may be cleared.
In the embodiment of the disclosure, when the target task is executed, the rollback data of the target task can be queried in the local data, and the target task is executed based on the rollback data without generating the rollback data, so that the rollback time of the target task is shortened, and the execution efficiency of the target task is improved.
Referring to fig. 2, a flowchart of another method for processing data in a database according to an embodiment of the disclosure is shown, where the method includes steps S201 to S211, where:
S201: and determining the time threshold and the polling time, and generating a configuration instruction.
S202: and determining whether the configuration instruction is configured successfully, if so, executing S203, and if not, executing S201.
S203: establishing the timing detection task;
s204: it is determined whether or not there is a timeout task identification, if so, S205 is executed, and if not, S206 is executed.
S205: waiting for the next timing detection task.
S206: and determining a target task corresponding to the task identifier, determining task information of the target task, and sending the task information to the execution object.
S207: it is determined whether the execution object is in an abnormal state, if so, S208 is executed, and if not, S209 is executed.
S208: and determining the other execution objects as the execution objects corresponding to the target tasks.
S209: and determining whether the analysis object is successfully analyzed, if so, executing S210, otherwise, returning an error to the execution module, and executing S207.
S210: and acquiring rollback data obtained by analyzing task execution information of the target task by the analysis object.
S211: and storing the rollback data into a user directory corresponding to the target task.
In summary, in the embodiment of the present disclosure, first, a target task with response timeout may be determined according to a task identifier, so as to monitor the task with response timeout, and after the target task is determined, task execution information of the target task may be determined, so as to reduce the data size of the task execution information that needs to be analyzed. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.
Based on the same inventive concept, the embodiments of the present disclosure further provide a data processing system of a database corresponding to a data processing method of the database, and since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to that of the data processing method of the database in the embodiments of the present disclosure, implementation of the system may refer to implementation of the method, and repeated parts will not be repeated.
Referring to fig. 3, a schematic frame structure of a data processing system of a database according to an embodiment of the disclosure is shown, where the system includes: the system comprises an operation and maintenance management platform 31, a management module 32, a gtid management module 33, a statement execution module 34 and an analysis module 35.
The operation and maintenance management platform 31 detects a configuration operation of task parameters for the timing detection task, where the task parameters include: and a time threshold, polling time and tenant id, and configuring the task parameters into the management module.
The management module 32 manages the timing detection task, detects the target task in the gtid management module based on the timing detection task, and controls the execution module to execute the pre-rollback operation, where the pre-rollback operation is used to indicate rollback data for determining the target task.
The gtid management module 33 generates a task identifier for the task in the distributed database system, and performs the preset operation on the task identifier.
Statement execution module 34 obtains task information for the target task, and determines an execution module based on the task information to send a pre-rollback instruction for the target task to the parsing module, where the pre-rollback instruction may be used to instruct execution of the pre-rollback operation.
The parsing module 35 determines task execution information of the target task in response to the pre-rollback instruction, and parses the task execution information to generate rollback data of the target task.
In the embodiment of the disclosure, firstly, a target task with response overtime can be determined according to the task identifier, so that monitoring of the task with response overtime is realized, and after the target task is determined, the task execution information of the target task can be determined, so that the data volume of the task execution information required to be analyzed is reduced. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
In a possible implementation manner, referring to fig. 4, a schematic operation process of the management module is shown, and specifically, the management module may detect a target task in the gtid management module based on a timing detection task. If the target task exists in the gtid module, the task information can be put back to the statement execution module, so that the statement execution module initiates a pre-rollback instruction and sends the pre-rollback instruction to the analysis module; if the target task does not exist in the gtid module, the target task in the gtid management module can be continuously detected based on the polling time and the timing detection task.
Based on the same inventive concept, the embodiments of the present disclosure further provide a data processing device of a database corresponding to a data processing method of the database, and since the principle of solving the problem of the device in the embodiments of the present disclosure is similar to that of the data processing method of the database in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.
Referring to fig. 5, a schematic diagram of a data processing apparatus of a database according to an embodiment of the disclosure is shown, where the apparatus includes: a first determination unit 51, a second determination unit 52, a generation unit 53; wherein, the liquid crystal display device comprises a liquid crystal display device,
A first determining unit 51, configured to determine a target task according to a task identifier, where the target task is a task that responds to a timeout;
a second determining unit 52 configured to determine task execution information of the target task;
and a generating unit 53, configured to generate rollback data of the target task based on the task execution information, so as to execute the target task based on the rollback data.
In the embodiment of the disclosure, firstly, a target task with response overtime can be determined according to the task identifier, so that monitoring of the task with response overtime is realized, and after the target task is determined, the task execution information of the target task can be determined, so that the data volume of the task execution information required to be analyzed is reduced. Next, rollback data for the target task may be generated based on the task execution information to execute the target task based on the rollback data. Based on the method, the target task responding to the overtime can be monitored, so that the data volume of task execution information required to be analyzed is reduced, and the time consumption for analyzing the task execution information is reduced. Meanwhile, as the target task with overtime response can be monitored, the condition that task execution information is cleaned due to too long occurrence time of the target task is also reduced, so that the success rate of rollback of the target task is improved, and the reliability of the distributed database system is further improved.
In a possible implementation manner, the first determining unit 51 is further configured to:
determining task execution time corresponding to the task identifier;
and under the condition that the task execution time exceeds a time threshold, determining the task corresponding to the task identifier as a target task.
In a possible implementation manner, the first determining unit 51 is further configured to:
determining the creation time of the task identifier;
and under the condition that the preset operation of the task identifier is not received, determining the task execution time corresponding to the task identifier based on the current moment and the creation time, wherein the preset operation is used for indicating ending the operation of the task corresponding to the task identifier.
In a possible implementation manner, the first determining unit 51 is further configured to:
in the case of receiving a configuration request, determining a time threshold indicated by the configuration request and a polling time;
and determining a monitoring operation for monitoring the task execution time based on the polling time, wherein the monitoring operation is used for indicating the time length of comparing the task execution time with a time threshold.
In a possible embodiment, the device is further configured to:
After determining a target task according to a task identifier, determining task information of the target task, wherein the task information comprises: a data storage location identifier and an execution object identifier;
determining an analysis object corresponding to the target task based on the task information; the analysis object is used for analyzing the task execution information and generating rollback data of the target task.
In a possible embodiment, the device is further configured to:
determining a data storage position corresponding to the target task based on the data storage position identifier;
and determining an execution object matched with the data storage position, and determining an analysis object corresponding to the data storage position based on the execution object.
In a possible embodiment, the second determining unit 52 is further configured to:
determining index information corresponding to the target task;
and searching task execution information corresponding to the task identifier in a task log based on the index information.
In a possible embodiment, the device is further configured to:
after the rollback data of the target task is generated, determining the storage state of the rollback data;
based on the storage state, under the condition that the rollback data is determined to be failed to be stored, establishing a storage period of the rollback data;
And executing storage operation on the rollback data according to the storage period.
In a possible implementation manner, the generating unit 53 is further configured to:
acquiring local data of the target task, and inquiring rollback data of the target task based on the local data;
determining operation sentences and task data of the target task based on the rollback data;
and executing the task data based on the operation statement.
The process flow of each unit in the apparatus and the interaction flow between units may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.
Corresponding to the data processing method of the database in fig. 1, the embodiment of the disclosure further provides a computer device 600, as shown in fig. 6, which is a schematic structural diagram of the computer device 600 provided in the embodiment of the disclosure, including:
a processor 61, a memory 62, and a bus 63; memory 62 is used to store execution instructions, including memory 621 and external memory 622; the memory 621 is also referred to as an internal memory, and is used for temporarily storing operation data in the processor 61 and data exchanged with the external memory 622 such as a hard disk, the processor 61 exchanges data with the external memory 622 through the memory 621, and when the computer device 600 is operated, the processor 61 and the memory 62 communicate with each other through the bus 63, so that the processor 61 executes the following instructions:
Determining a target task according to the task identification, wherein the target task is a task responding to overtime;
determining task execution information of the target task;
and generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the data processing method of the database described in the method embodiments above. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The embodiments of the present disclosure further provide a computer program product, where the computer program product carries a program code, where instructions included in the program code may be used to perform steps of a data processing method of a database described in the foregoing method embodiments, and specifically reference may be made to the foregoing method embodiments, which are not described herein.
Wherein the above-mentioned computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the foregoing examples are merely specific embodiments of the present disclosure, and are not intended to limit the scope of the disclosure, but the present disclosure is not limited thereto, and those skilled in the art will appreciate that while the foregoing examples are described in detail, it is not limited to the disclosure: any person skilled in the art, within the technical scope of the disclosure of the present disclosure, may modify or easily conceive changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features thereof; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the disclosure, and are intended to be included within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (12)
1. A method of processing data in a database, comprising:
determining a target task according to the task identification, wherein the target task is a task responding to overtime;
determining task execution information of the target task;
and generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
2. The method of claim 1, wherein the determining the target task based on the task identification comprises:
determining task execution time corresponding to the task identifier;
and under the condition that the task execution time exceeds a time threshold, determining the task corresponding to the task identifier as a target task.
3. The method of claim 2, wherein the determining the task execution time corresponding to the task identity comprises:
determining the creation time of the task identifier;
and under the condition that the preset operation of the task identifier is not received, determining the task execution time corresponding to the task identifier based on the current moment and the creation time, wherein the preset operation is used for indicating ending the operation of the task corresponding to the task identifier.
4. The method according to claim 2, wherein the method further comprises:
in the case of receiving a configuration request, determining a time threshold indicated by the configuration request and a polling time;
and determining a monitoring operation for monitoring the task execution time based on the polling time, wherein the monitoring operation is used for indicating the time length of comparing the task execution time with a time threshold.
5. The method according to claim 1, wherein the method further comprises:
after determining a target task according to a task identifier, determining task information of the target task, wherein the task information comprises: a data storage location identifier and an execution object identifier;
determining an analysis object corresponding to the target task based on the task information; the analysis object is used for analyzing the task execution information and generating rollback data of the target task.
6. The method of claim 5, wherein determining, based on the task information, a resolution object corresponding to the target task comprises:
determining a data storage position corresponding to the target task based on the data storage position identifier;
And determining an execution object matched with the data storage position, and determining an analysis object corresponding to the data storage position based on the execution object.
7. The method of claim 1, wherein determining task execution information for the target task comprises:
determining index information corresponding to the target task;
and searching task execution information corresponding to the task identifier in a task log based on the index information.
8. The method according to claim 1, wherein the method further comprises:
after the rollback data of the target task is generated, determining the storage state of the rollback data;
based on the storage state, under the condition that the rollback data is determined to be failed to be stored, establishing a storage period of the rollback data;
and executing storage operation on the rollback data according to the storage period.
9. The method of claim 1, wherein the performing the target task based on the rollback data comprises:
acquiring local data of the target task, and inquiring rollback data of the target task based on the local data;
determining operation sentences and task data of the target task based on the rollback data;
And executing the task data based on the operation statement.
10. A data processing apparatus of a database, comprising:
the first determining unit is used for determining a target task according to the task identification, wherein the target task is a task responding to overtime;
the second determining unit is used for determining task execution information of the target task;
and the generating unit is used for generating rollback data of the target task based on the task execution information so as to execute the target task based on the rollback data.
11. A computer device, comprising: a processor, a memory and a bus, said memory storing machine readable instructions executable by said processor, said processor and said memory communicating via the bus when the computer device is running, said machine readable instructions when executed by said processor performing the steps of the data processing method of the database according to any of claims 1 to 9.
12. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when run by a processor, performs the steps of the data processing method of a database according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211645517.7A CN116186030A (en) | 2022-12-16 | 2022-12-16 | Data processing method and device of database and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211645517.7A CN116186030A (en) | 2022-12-16 | 2022-12-16 | Data processing method and device of database and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116186030A true CN116186030A (en) | 2023-05-30 |
Family
ID=86441270
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211645517.7A Pending CN116186030A (en) | 2022-12-16 | 2022-12-16 | Data processing method and device of database and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116186030A (en) |
-
2022
- 2022-12-16 CN CN202211645517.7A patent/CN116186030A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10152382B2 (en) | Method and system for monitoring virtual machine cluster | |
| US8595556B2 (en) | Soft failure detection | |
| CN107688531B (en) | Geo-database integration test method, device, computer equipment and storage medium | |
| JP6333410B2 (en) | Fault processing method, related apparatus, and computer | |
| CN110442498B (en) | Abnormal data node positioning method and device, storage medium and computer equipment | |
| US9841986B2 (en) | Policy based application monitoring in virtualized environment | |
| CN106789306B (en) | Method and system for detecting, collecting and recovering software fault of communication equipment | |
| CN111274059B (en) | Software exception handling method and device of slave device | |
| KR20110091776A (en) | System for assisting with execution of actions in response to detected events, method for assisting with execution of actions in response to detected events, assisting device, and computer program | |
| CN111177165B (en) | Method, device and equipment for detecting data consistency | |
| CN105573859A (en) | Data recovery method and device of database | |
| CN110609778A (en) | Method and system for storing server downtime log | |
| CN111046011A (en) | Log collection method, system, node, electronic device and readable storage medium | |
| CN109753378A (en) | A kind of partition method of memory failure, device, system and readable storage medium storing program for executing | |
| CN108958965A (en) | A kind of BMC monitoring can restore the method, device and equipment of ECC error | |
| CN109586989A (en) | A kind of state detection method, device and group system | |
| CN109491856B (en) | Bus monitoring system, method and device | |
| CN109522184A (en) | A kind of server system method for safety monitoring, device and terminal | |
| CN112069032A (en) | Availability detection method, system and related device for virtual machine | |
| CN116186030A (en) | Data processing method and device of database and computer equipment | |
| JP2018180982A (en) | Information processing device and log recording method | |
| CN115314361B (en) | Server cluster management method and related components thereof | |
| CN111159051A (en) | Deadlock detection method and device, electronic equipment and readable storage medium | |
| CN112988442B (en) | Method and equipment for transmitting fault information in server operation stage | |
| CN114116330A (en) | Server performance test method, system, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |