CN112749156A

CN112749156A - Data processing method, database management system and data processing equipment

Info

Publication number: CN112749156A
Application number: CN201911040327.0A
Authority: CN
Inventors: 于巍; 李国良; 张金玉; 董亚辉
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-10-29
Filing date: 2019-10-29
Publication date: 2021-05-04

Abstract

The embodiment of the application discloses a data processing method which is used for retry of execution failure of a storage process. The method comprises the following steps: executing a storage process in response to a storage process request, wherein the storage process comprises one or more ordered transactions, and when the execution of a first transaction in the one or more ordered transactions is finished, recording mark information of the first transaction; and retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction based on the condition that the exception occurs in the storage process and the retrying of the storage process is met.

Description

Data processing method, database management system and data processing equipment

Technical Field

The present application relates to the field of database technologies, and in particular, to a data processing method, a database management system, and a data processing device.

Background

The stored procedure (stored procedure) is a set of statements capable of completing a specific function, which is stored in a database system, and is called again after first compiling without being compiled again, and a user can run the stored procedure by specifying the name of the stored procedure. The storage process retry is a retry mechanism when the execution of the storage process fails, and can avoid interruption of user services caused by hardware failure or software failure. A transaction (transaction) is a sequence of operations performed as a single logical unit of work in a database system, which are either all performed or not performed at all, and is an indivisible unit of work. For example, in a relational database, a transaction may be a Structured Query Language (SQL) statement, a set of SQL statements, or an entire program. A stored procedure may contain one or more transactions.

In the prior art, a database driver calls a specified storage process according to a work request sent by a user to respond to the request of the user, and when the execution of the storage process fails and the failure reason is recoverability abnormality, the database driver sends a retry request to a Database (DB) kernel, acquires a correct processing result, and then responds to the user. Therefore, partial hardware and software errors can be shielded, and user experience is improved.

In the prior art, the database driver acquires whether the called storage process is successfully executed or not, and when the storage process is unsuccessfully executed, the retry can only re-execute the whole storage process once. When a transaction commit instruction exists in the storage process, the whole retry of the storage process causes the partial operation which is already committed to be repeatedly executed, thereby causing data errors.

Disclosure of Invention

The embodiment of the application provides a data processing method, which is used for retry of execution failure in a storage process and can avoid repeated execution of executed transactions in the storage process.

A first aspect of an embodiment of the present application provides a data processing method, including: executing a storage process in response to a storage process request, wherein the storage process comprises one or more ordered transactions, and when the execution of a first transaction in the one or more ordered transactions is finished, recording mark information of the first transaction; and based on the exception occurring in the storage process and the condition of retrying the storage process is met, retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction.

The storing process may contain one or more ordered transactions, and during the execution of the storing process, marking information of executed transactions in the storing process, such as the position of the end of the transaction, is recorded. Therefore, when the execution of the storage process fails to retry, only the statements after the executed transaction can be retried, and the executed transaction does not need to be re-executed, so that the execution efficiency of retry is improved, and errors caused by repeatedly executing the submitted transaction during retry are avoided.

In a possible implementation manner of the first aspect, the method further includes: the request statement that called the stored procedure is recorded.

The data processing method provided by the embodiment of the application records a request statement for calling a storage process, and can be used for positioning the position of retry execution through the request statement and the mark information of the first transaction.

In one possible implementation form of the first aspect, the marking information includes a line number of an end statement of the first transaction.

The data processing method provided by the embodiment of the application provides a specific implementation mode for recording the marking information of the first transaction, and enhances the operability of the scheme.

In a possible implementation manner of the first aspect, the method further includes: recording intermediate state information of the stored procedure, the intermediate state information including one or more of: the execution statement result of the first transaction, error prompt information, input parameters, output parameters, global variables, and temporary variables.

In the data processing method provided by the embodiment of the application, the database management system may further record an execution statement result or error prompt information, an input parameter, an output parameter, a global variable, and a temporary variable of the first transaction, and retry is performed according to the intermediate state information when the first transaction is ended, so that data consistency can be satisfied.

In a possible implementation manner of the first aspect, the intermediate state information includes an execution statement result or error hint information of the first transaction, and the method further includes: and caching the execution statement result or the error prompt message of the first transaction until the first transaction is ended, and then sending the execution statement result or the error prompt message to the user equipment.

According to the data processing method provided by the embodiment of the application, the execution statement result or the error prompt information of the first transaction is recorded and sent to the user equipment after the first transaction is ended, so that the same execution statement result or the same error prompt information can be prevented from being repeatedly sent to the user equipment when retry is avoided.

In a possible implementation manner of the first aspect, the intermediate state information includes an execution statement result or error hint information of the first transaction, and the method further includes: and caching the execution statement result or the error prompt message of the first transaction until the storage process is finished, and then sending the execution statement result or the error prompt message to the user equipment.

The data processing method provided by the embodiment of the application provides another way for sending the execution statement result or the error prompt information to the user, namely, the user is sent after the storage process is finished, so that the same execution statement result or the same error prompt information can be prevented from being repeatedly sent to the user equipment when the user equipment retries.

In a possible implementation manner of the first aspect, the first transaction is located at a branch of a branch statement in the storing process, and the method further includes: recording intermediate state information of the stored process, wherein the intermediate state information comprises a branch execution state, and the branch execution state is used for identifying a branch where the first transaction is located; retrying transactions subsequent to the first transaction in the storage process includes: based on the branch execution state, transactions subsequent to the first transaction in the branch in which the first transaction is located are executed.

In the data processing method provided in the embodiment of the present application, the intermediate state information may further include a branch execution state, which is used to identify a branch of the first transaction in a branch statement, where the branch statement includes an if branch statement or a case branch statement. When the retry is carried out, the branch can be continuously executed according to the branch execution state, so that the retry error is avoided.

In a possible implementation manner of the first aspect, the branch execution state includes a conditional result of executing the branch statement in the storage process, and the conditional result is used to determine to execute a target branch in the branch statement.

According to the data processing method provided by the embodiment of the application, one specific record content of the branch execution state is a conditional judgment result of a branch statement, so that the realizability of the scheme is enhanced.

In a possible implementation manner of the first aspect, the method further includes: the condition for retrying the stored procedure is that the stored procedure exception is a recoverability exception, the recoverability exception including: network exceptions, memory usage exceptions, input output exceptions, lock wait timeout exceptions, or two-phase transaction commit exceptions.

According to the data processing method provided by the embodiment of the application, if the exception is a recoverability exception, retry can be executed.

In a possible implementation manner of the first aspect, the retrying the transaction after the first transaction in the storing process includes: and after waiting for the preset first time length, retrying the transaction after the first transaction in the storage process.

The data processing method provided by the embodiment of the application can improve the retry success rate by retrying after waiting for the preset duration in the scenes such as the master-slave switching.

In a possible implementation manner of the first aspect, the method further includes: when the execution of a second transaction in the one or more ordered transactions is finished, recording the marking information of the second transaction, wherein the second transaction is executed after the first transaction; the retrying of transactions subsequent to the first transaction in the storage process includes: and retrying the transactions after the second transaction in the storage process according to the marking information of the second transaction, wherein the second transaction is the last executed transaction in the executed transactions when the condition of retrying the storage process is met.

According to the data processing method provided by the embodiment of the application, if the storage process comprises a plurality of transactions, the mark information of each transaction can be recorded, and when the transaction is retried, the transaction after the last executed transaction is executed, so that the resource waste can be further reduced.

A second aspect of the embodiments of the present application provides a database management system, including: an execution unit to execute a stored procedure in response to a stored procedure request, the stored procedure comprising one or more ordered transactions; the recording unit is used for recording mark information of a first transaction in the one or more ordered transactions when the execution of the first transaction is finished; and the retry unit is used for retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction based on the condition that the exception occurs in the storage process and the retrying of the storage process is met.

In a possible implementation manner of the second aspect, the recording unit is further configured to: the request statement that called the stored procedure is recorded.

In one possible implementation of the second aspect, the marker information includes a line number of an end statement of the first transaction.

In one possible implementation manner of the second aspect, the database management system further includes: a recording unit, configured to record intermediate state information of the stored procedure, where the intermediate state information includes one or more of: the execution statement result of the first transaction, error prompt information, input parameters, output parameters, global variables, and temporary variables.

In a possible implementation manner of the second aspect, the intermediate state information includes an execution statement result or error prompt information of the first transaction, and the database management system further includes: and the cache unit is used for caching the execution statement result or the error prompt information of the first transaction until the first transaction is ended, and then sending the execution statement result or the error prompt information to the user equipment.

In a possible implementation manner of the second aspect, the intermediate state information includes an execution statement result or error prompt information of the first transaction, and the database management system further includes: and the cache unit is used for caching the execution statement result or the error prompt information of the first transaction until the storage process is finished, and then sending the execution statement result or the error prompt information to the user equipment.

In a possible implementation manner of the second aspect, the first transaction is located at a branch of a branch statement in the storing process, and the recording unit is further configured to: recording intermediate state information of the stored process, wherein the intermediate state information comprises a branch execution state, and the branch execution state is used for identifying a branch where the first transaction is located; the retry unit is specifically configured to execute, based on the branch execution status, a transaction subsequent to a first transaction in a branch in which the first transaction is located.

In a possible implementation manner of the second aspect, the branch execution state includes a conditional result of executing the branch statement in the storage process, and the conditional result is used to determine to execute a target branch in the branch statement.

In one possible implementation of the second aspect, the condition for retrying the storage process is that the storage process exception is a recoverability exception, the recoverability exception including: network exceptions, memory usage exceptions, input output exceptions, lock wait timeout exceptions, or two-phase transaction commit exceptions.

In a possible implementation manner of the second aspect, the retry unit is specifically configured to: and after waiting for the preset first time length, retrying the transaction after the first transaction in the storage process.

In a possible implementation manner of the second aspect, the recording unit is further configured to, when execution of a second transaction of the one or more ordered transactions is finished, record marking information of the second transaction, where the second transaction is a transaction executed after the first transaction; the retry unit is specifically configured to: and retrying the transactions after the second transaction in the storage process according to the marking information of the second transaction, wherein the second transaction is the last executed transaction in the executed transactions when the condition of retrying the storage process is met.

A third aspect of the embodiments of the present application provides a data processing apparatus, including: a memory to store instructions; a processor configured to execute the instructions in the memory to cause the data processing apparatus to perform the steps of the foregoing first aspect or each implementation manner of the first aspect.

A fourth aspect of embodiments of the present application provides a computer program product, which is characterized in that the computer program product includes instructions that, when executed on a computer, cause the computer to execute the steps of the foregoing first aspect or each implementation manner of the first aspect.

A fifth aspect of embodiments of the present application provides a computer-readable storage medium, which stores instructions that, when executed on a computer, cause the computer to perform the steps of the foregoing first aspect or each implementation manner of the first aspect.

A sixth aspect of the present embodiment provides a database system, including a network interface, configured to receive a storage procedure request sent by a user equipment; a data processing apparatus configured to perform the method according to the first aspect or any of the implementations of the first aspect in response to the stored procedure request.

According to the technical scheme, the embodiment of the application has the following advantages: the data processing method provided by the embodiment of the application realizes the execution failure retry of the storage process, supports the execution failure retry of the storage process containing the transaction, improves the execution efficiency of the retry because the finished transaction does not need to be executed again, and avoids errors caused by repeatedly executing the executed transaction during retry.

Drawings

FIG. 1a is a diagram illustrating an architecture of a stand-alone database system according to an embodiment of the present application;

FIG. 1b is a schematic diagram of an architecture of a clustered database system according to an embodiment of the present application;

FIG. 1c is a schematic diagram of another architecture of a clustered database system according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a database server in an embodiment of the present application;

FIG. 3 is a schematic diagram of an embodiment of a data processing method in an embodiment of the present application;

FIG. 4 is a schematic diagram of an embodiment of a process for creating a store in an embodiment of the application;

FIG. 5 is a flow chart illustrating a stored procedure execution process according to an embodiment of the present application;

FIG. 6 is a system architecture diagram of a cluster scenario in an embodiment of the present application;

FIG. 7 is a diagram of an embodiment of retry of a storage process in a cluster scenario in an embodiment of the present application;

FIG. 8 is a diagram of another embodiment of retry of a storage process in a cluster scenario in an embodiment of the present application;

FIG. 9 is a system architecture diagram illustrating a retry method for failed execution of a stored procedure according to an embodiment of the present application;

FIG. 10 is a schematic diagram of an embodiment of a database management system in an embodiment of the present application;

FIG. 11 is a schematic diagram of another embodiment of a database management system in an embodiment of the present application;

fig. 12 is a schematic diagram of an embodiment of a data processing apparatus in the embodiment of the present application.

Detailed Description

The embodiment of the application provides a data processing method, which is used for retry of execution failure in a storage process and can avoid repeated execution of transactions in the storage process.

Embodiments of the present application will now be described with reference to the accompanying drawings, and it is to be understood that the described embodiments are merely illustrative of some, but not all, embodiments of the present application. As can be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.

The terms "first," "second," and the like in the description and in the claims of the present application and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Moreover, the terms "comprises," "comprising," and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not expressly listed or inherent to such process, method, article, or apparatus. The naming or numbering of the steps appearing in the present application does not mean that the steps in the method flow have to be executed in the chronological/logical order indicated by the naming or numbering, and the named or numbered process steps may be executed in a modified order depending on the technical purpose to be achieved, as long as the same or similar technical effects are achieved.

The data processing method, the database management system and the database server provided by the embodiment of the application can be applied to a stand-alone database system or a clustered database system, and are not limited herein. A database system (database system) is a data processing system developed to meet the requirement of data processing, please refer to fig. 1a-1c, which are schematic diagrams of the database system in the embodiment of the present application, and the following is briefly introduced:

fig. 1a is a schematic diagram of an architecture of a stand-alone database system in an embodiment of the present application, where the stand-alone database system includes a database management system and a data store (data store), the database management system is used for providing services such as query and modification of a database, and the database management system stores data in the data store. In a stand-alone database system, the database management system and data store are typically located on a single server, such as a symmetric multi-processor (SMP) server. The SMP server includes multiple processors, all of which share resources, such as buses, memory, and i/o systems. The functions of the database management system may be implemented by one or more processors executing programs in memory.

Fig. 1b is a schematic structural diagram of a cluster database system in an embodiment of the present application, where the cluster database system is a cluster database system adopting a shared-storage architecture, and includes a plurality of nodes (e.g., nodes 1 to N in the figure), each node is deployed with a database management system, and provides services such as query and modification of a database for a user, and the plurality of database management systems store shared data in a shared data storage and perform read-write operations on the data in the data storage through a switch. The shared data storage may be a shared disk array. The nodes in the clustered database system may be physical machines, such as database servers, or virtual machines running on abstracted hardware resources. If the node is a physical machine, the switch is a Storage Area Network (SAN) switch, an ethernet switch, a fiber switch, or other physical switching device. And if the node is a virtual machine, the switch is a virtual switch.

Fig. 1c is a schematic diagram of another architecture of a clustered database system in an embodiment of the present application, where the clustered database system is a clustered database system adopting a shared-nothing architecture, and includes a plurality of nodes (e.g., nodes 1 to N in the figure), each node having a respective hardware resource (e.g., a data storage), an operating system, and a database, and the nodes communicate with each other through a network. Under the system, data are distributed to each node according to the database model and application characteristics, a query task is divided into a plurality of parts, the parts are executed in parallel on all nodes and are calculated in a mutual cooperation mode to provide database service as a whole, and all communication functions are realized on a high-broadband network interconnection system. Like the clustered database system of shared-storage architecture described in the clustered database system 1, the nodes herein may be physical machines or virtual machines.

The nodes in the clustered database system shown in fig. 1b and 1c may be configured as coordinator nodes (coordinators) or data nodes (data nodes) according to actual usage requirements, the coordinator nodes are configured to receive SQL or stored procedure requests sent by clients, and after analysis, the SQL may be sent to data nodes that need to participate in data processing. And the data node finishes data extraction, the query result is returned to the coordination node, and the coordination node aggregates the data and sends the aggregated data to the client.

In all embodiments herein, the data store of the database system includes, but is not limited to, a Solid State Disk (SSD), a disk array, or other type of non-transitory computer-readable medium. Although the database is not shown in fig. 1a-1c, it should be understood that the database is stored in a data store. Those skilled in the art will appreciate that a database system may include fewer or more components than shown in fig. 1a-1c, or different components than those shown in fig. 1a-1c, with fig. 1a-1c merely illustrating components that are more relevant to the disclosed implementation of the present embodiments. For example, although 4 nodes have been described in both the clustered database systems of FIGS. 1b and 1c, one skilled in the art will appreciate that a clustered database system may contain any number of nodes. The database management system functions of each node may be implemented by a suitable combination of software, hardware, and/or firmware running on each node, respectively.

For convenience of understanding and description, the solution of the embodiment of the present application is described below by taking a node in a shared-nothing architecture clustered database system, such as a database server, as an example and not a limitation. It will be apparent to those skilled in the art from this disclosure that the methods of the embodiments of the present application can be applied to clustered database systems of shared-storage architecture, stand-alone database systems, and any type of relational database system.

Please refer to fig. 2, which is a schematic structural diagram of a database server in an embodiment of the present application.

As shown in fig. 2, an embodiment of the present application provides a database server 100, including: at least one processor 104, a non-transitory computer-readable medium 106 storing executable code, and a database management system 108. The executable code, when executed by the at least one processor 104, is configured to implement the components and functions of the database management system 108. The non-transitory computer-readable medium 106 may include one or more non-volatile memories, including by way of example semiconductor memory devices such as Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), and flash memory (flash memory); magnetic disks such as internal hard disks (internal hard disks) or removable disks (removable disks), magneto-optical disks (magnetic optical disks), and CD-ROMs and DVD-ROMs. Furthermore, non-transitory computer-readable medium 106 may also include any device configured as a main memory (main memory). The at least one processor 104 may comprise any type of general purpose computing circuitry or special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The at least one processor 104 may also be one or more processors, such as CPUs, coupled to one or more semiconductor substrates.

The database management system 108 may be a relational database management system (RDBMS). Database management system 108 supports Structured Query Language (SQL). Generally, SQL refers to a specialized programming language that is dedicated to managing data stored in a relational database. SQL may refer to various types of data-related languages, including, for example, a data definition language and a data manipulation language, where the scope of SQL may include data insertion, querying, updating and deletion, schema creation and modification, and data access control. Further, in some examples, SQL may include descriptions related to various language elements.

A query (query) is a request to view, access and/or manipulate data stored in a database. The database management system 108 may receive a query in SQL format (referred to as an SQL query) from the database client 102. In general, the database management system 108 generates query results corresponding to a query by accessing and manipulating relevant data from the database and returning the query results to the database client 102. A database is a collection of data organized, described, and stored in a mathematical model, and may include one or more database structures or formats, such as row stores and column stores. The database is typically stored in a data store, such as external data store 120 in fig. 2, or non-transitory computer readable medium 106. When a database is stored in the non-transitory computer readable medium 106, the database management system 108 is an in-memory database management system.

The database client 102 may include any type of device or application configured to interact with the database management system 108. In some examples, database client 102 includes one or more application servers.

The database management system 108 includes an SQL engine 110, an execution engine 122, and a storage engine 134. The SQL engine 110 generates a corresponding execution plan according to an SQL statement submitted by the client 102, such as a Query (Query), and the execution engine 122 operates according to the execution plan of the statement to generate a Query result. The storage engine 134 is responsible for managing the actual contents of the data and index of the table on the file system, and also manages the data such as Cache, Buffer, transaction, Log and the like during the operation. For example, the storage engine 134 may write the results of execution by the execution engine 122 to the data store 120 via physical I/O. The SQL engine 110 includes a parser (not shown) for performing syntactic and semantic analysis of SQL statements, expanding the view in the query, and dividing it into small query blocks, and an optimizer (not shown). The optimizer generates a set of execution plans that may be used for the statements, estimates the cost of each execution plan, compares the costs of the plans, and finally selects an execution plan with the smallest cost.

The following is an introduction of important concepts related to embodiments of the present application:

the stored procedure (stored procedure) is a set of statements capable of completing a specific function, which is stored in a database system, and is called again after first compiling without being compiled again, and a user can run the stored procedure by specifying the name of the stored procedure.

A transaction (transaction) is a sequence of database operations defined by a user in a database system, which are either all executed or not executed at all, and are an indivisible unit of work. A transaction has four properties: atomicity (atomic), consistency (consistency), isolation (isolation), and durability (durability). In a relational database, a transaction may be a Structured Query Language (SQL) statement, a set of SQL statements, or an entire program. Statements in SQL that define transactions include: BEGIN, COMMIT, ROLLBACK.

Transactions typically BEGIN with a BEGIN and end with a COMMIT or ROLLBACK. The COMMIT represents a COMMIT, i.e., all operations that COMMIT the transaction, specifically, all updates to the database in the transaction are written to disk, and the transaction ends normally. ROLLBACK means that when some failure occurs during the operation of the transaction, the transaction cannot continue to execute, and the system completely withdraws (UNDO) all completed operations on the database in the transaction, and rolls back to the state at the beginning of the transaction.

In the prior art, a database server executes a storage process according to a request of a client, when the execution of the storage process fails, only the whole storage process can be retried completely, and the control granularity is the storage process level.

When a storage process comprises one or more transactions, if a fault occurs in the storage process, a transaction ends with a COMMIT, which means that all updates to the database in the transaction have been written to the disk. When the storage process is retried, the transaction is repeatedly executed, resulting in data errors. If a transaction ends with a ROLLBACK before an error occurs in the storage process, the transaction cannot be continuously executed, and when the storage process is retried, the ROLLBACK transaction is repeatedly executed, so that resources are wasted.

In addition, retry in the prior art needs to consider driving support of different systems, such as open database connectivity (ODBC), Java database connectivity (JDBC), c language application program interface (Libpq) of object-relational database management system (ordms) postgresql, and the like.

In order to solve the above problem, embodiments of the present application provide a data processing method for performing a retry procedure of a storage procedure, which is described below with reference to specific embodiments.

Please refer to fig. 3, which is a diagram illustrating a data processing method according to an embodiment of the present application.

301. Executing a storage process;

a database management system receives a storage process request sent by a client; the stored procedure can be created by a client, please refer to fig. 4, and to illustrate an embodiment of creating the stored procedure, a statement for creating the stored procedure is sent by an SQL client, and the statement is registered in the database. Thereafter, the stored procedure may be invoked directly, for example, by the statement "Call STPTest1 (); "send a stored procedure request to implement the call, execute the stored procedure in the database. The storage process includes one or more ordered transactions.

302. Recording a first position and intermediate state information of the end of the first transaction;

during the execution of the storage process, when a transaction is finished, the database management system may record the mark information of the transaction, and optionally, the database management system records the position information of the statement of the end of the transaction, or the identification of the transaction. Specifically, the database management system records the line number of the transaction end statement. The location at the end of the transaction, including the location of COMMIT or the location of ROLLBACK. For example, as shown in fig. 4, if the fourth line fails, the last COMMIT position before the fourth line, i.e., the second line (line 2), is recorded;

in addition, the database management system also records intermediate state information of the stored procedures.

The intermediate state information includes various parameters or variables generated during the execution of the stored procedures, such as the result of execution statements, error prompt information generated during execution, temporary variables, global variables, input parameters, output parameters, branch execution states, and the like. It is understood that the intermediate state information is data on which retry must depend, and data consistency cannot be satisfied if the intermediate state information is missing during retry.

Optionally, the database management system caches the result of the executed statement or the error prompt information generated during execution. In the storage process, if an intermediate result needs to be returned before the COMMIT, the intermediate result is firstly cached and is not sent to the user equipment until the COMMIT is sent, or the intermediate result is sent uniformly when the execution of the storage process is finished. Illustratively, the statement "rain noise 'hello world'; the prompt information 'hello world' for indicating screen printing is directly sent before the transaction is finished and also sent again when the transaction is retried, so that the same information is printed on the screen for a plurality of times, and therefore, the execution statement result is stored through data buffering and is sent when the execution of the storage process is finished or the transaction is finished, and repeated operation can be avoided.

Optionally, the database management system records values of variables of the stored process, including temporary variable execution results. For example, after the second row COMMIT in FIG. 4 is executed, the value of the variable i at that time is recorded.

Optionally, the database management system records input parameters and/or output parameters of the stored procedure, for example, input parameters x, y of the stored procedure in fig. 4.

Optionally, the first transaction is located in a branch of the branch statement in the storing process, and the database management system records a branch execution state, which is used to identify the branch where the first transaction is located. The branch execution state includes a conditional branch execution state or a case branch execution state, and the like. Optionally, the branch execution state is a conditional result of the branch statement. Please refer to fig. 5, which is a flowchart illustrating a process of executing a storage procedure in an embodiment of the present application. For example, as shown in fig. 5, if the if branch is included in the execution process of the storage process, the branch determination result needs to be recorded, for example, if the if (select) is 6, the first branch is entered if the select is equal to 6, and the second branch is entered if the select is not equal to 6; recording that the branch judgment result is equal to 6, which can be used for determining to execute the first branch, and recording that the branch judgment result is not equal to 6, which can be used for determining to execute the second branch, and when the storage process retries, the branch determined according to the branch judgment result can be directly entered without re-executing the if judgment statement.

For a storage process containing multiple transactions, there is corresponding intermediate state information for the end location of each transaction. For example, if the storage process includes 3 transactions, which are an a transaction, a B transaction, and a C transaction according to the execution sequence, in the execution process of the storage process, when the a transaction ends, corresponding first intermediate state information is recorded, and when the B transaction ends, corresponding second intermediate state information is recorded, at this time, optionally, the first intermediate state information may be released, and if an error occurs in the execution process of the C transaction, when a retry is performed, the already ended a transaction and B transaction may be skipped, and the retry is performed after the end position of the B transaction according to the second intermediate state information.

Optionally, recording a request statement for calling the stored procedure sent by the client, such as "Call STPTest1 (); "; the request statement and the transaction end statement line number may be used together to determine where the retry begins.

303. Retrying the stored procedure according to a first location where the first transaction ended and the intermediate state information;

and based on the exception occurring in the storage process and the condition of retrying the storage process is met, retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction.

The condition for retrying the storage process includes the exception being a recoverable exception. There are various types of exceptions that cause a failure in the execution of a stored procedure, such as: network exceptions such as network flash, memory usage exceptions such as memory application failure, input/output (I/O) exceptions such as read exceptions due to hard disk busy, lock wait timeout exceptions, and exceptions that occur during two-phase transaction execution, including two-phase transaction commit exceptions. The specific reason for the failure of the execution of the stored procedure is not limited herein. If the exception is a recoverable exception that can be resolved by a retry, a retry of the storage process will be initiated.

The database management system can realize exception capture, identify exception types, judge whether the exception is a recoverable exception or not to determine whether retry is needed or not, and initiate retry logic at the server side if retry is needed. The specific process is not described here.

It should be noted that, in the aspect of data processing in the embodiment of the present application, the retry of failed execution of the storage process in different database usage scenarios may be applicable, and the retry of failed execution of the storage process in different database usage scenarios includes application scenarios such as a stand-alone database, a cluster database, and a master-slave database.

The retry includes one retry or multiple retries, the retry timing may be a direct retry, a retry after waiting for a preset time period, or a retry when a preset condition is met, and the specific retry number and the retry timing are not limited herein.

And retrying the transactions after the first transaction in the storage process according to the marking information of the first transaction. When the database management system retries, the executed and submitted statements are skipped through the execution intermediate state recorded before, the consistency state after the previous submission is restored, and the transaction after the first transaction is retried.

It is understood that the storage process may include one or more transactions, and if there are multiple transactions in a storage process, when the execution of the storage process fails, the retry may be performed after the last transaction that ends, for example, for a storage process containing 3 transactions, if there is an error in the execution of the 3 rd transaction, the retry may be performed after the 2 nd transaction ends.

Illustratively, as shown in fig. 4, if the fourth row fails, the retry will skip the statements before the COMMIT of the second row, and continue to execute the statements after the second row according to the variable i, the input parameters x and y, the intermediate result generated by the execution, and the like recorded in step 302.

If the retry execution is successful, a correct execution result can be returned to the user.

The data processing method provided by the embodiment of the application realizes the execution failure retry of the storage process, supports the execution failure retry of the storage process containing the transaction, and improves the execution efficiency of the retry because the finished transaction does not need to be re-executed and the retry granularity is reduced to the transaction level, thereby avoiding the error caused by repeatedly executing the submitted part during the retry.

Fig. 6 is a schematic diagram of a system architecture of a cluster scene in the embodiment of the present application.

In a database cluster scenario, the database cluster scenario includes a coordinating node (coordinator) and a plurality of data nodes (data nodes), where the data nodes shown in fig. 6 are deployed in a master-slave manner, and when a fault occurs in a master data node, the backup data node takes over an inquiry request. When a fault occurs, the coordinating node can decide whether to retry or not, retry strategies and the like according to the retry decision module. The embodiment of the application can be used for processing the retry of the primary and standby switching scene and dealing with various conditions such as planned halt, unplanned halt and the like. Typically, the retry of the execution failure of the storage process and the retry of the execution failure scenario caused by the switching of the main/standby nodes are included. The following are introduced separately:

firstly, in a cluster scene, the retry of the execution failure of the storage process.

Please refer to fig. 7, which is a diagram illustrating an embodiment of retry of a storage process in a cluster scenario in the embodiment of the present application.

701. The service application sends a storage process request to the coordination node;

in a cluster database scenario, a business application sends a storage process request to a coordinating node.

702. The coordinating node sends the storage process request;

the coordination node executes the storage process request, sends SQL to the data node according to actual requirements or preset rules, and executes partial SQL by the data node.

The coordinating node can acquire the intermediate state information in the storage process and can capture the exception in the execution process.

703. Synchronizing the main and standby devices;

if the data nodes are configured in a master-standby mode, the SQL received by the group data nodes is synchronized to the standby data nodes.

It should be noted that step 703 is an optional step, and may or may not be executed, and is not limited herein.

704. The coordination node acquires failure information;

the storage process execution failure comprises the coordination node execution failure or the data node execution failure.

Optionally, there are various types of exceptions that cause the execution of the storage process to fail, for example: network exceptions such as network flash, memory usage exceptions such as memory application failure, input/output (I/O) exceptions such as read exceptions due to hard disk busy, lock wait timeout exceptions, and exceptions that occur during two-phase transaction execution. The specific reason for the failure of the execution of the stored procedure is not limited herein. If the exception is a recoverable exception, a retry of the storage process is initiated.

Optionally, the data node executes the SQL and returns an execution result to the coordination node, and if the data node executes an error, the coordination node may obtain failure information of the storage process.

For a specific implementation process, please refer to step 302 in the embodiment corresponding to fig. 3, which is not described herein again.

705. The coordinating node checks the failure reason;

and the coordination node identifies whether the reason of the data node execution failure is a recoverability abnormity which can be recovered through retry or not, and if so, initiates retry. Optionally, the coordination node decision module determines whether to retry or not and the time for which retry needs to be delayed according to the current master/slave condition.

706. The storage procedure is retried.

The coordinating node sends a retry request according to the recorded intermediate state information in the storage process, and the primary data node or the backup data node performs retry according to the intermediate state information in the storage process, where the specific implementation process refers to step 303 in the embodiment corresponding to fig. 3, and details are not described here.

And secondly, under the cluster scene, the master node and the standby node are switched to cause retry of execution failure.

Please refer to fig. 8, which is a diagram illustrating another embodiment of retry of the storage process in the cluster scenario in the embodiment of the present application.

801. The service application sends a storage process request to the coordination node;

802. the coordinating node sends the storage process request;

803. synchronizing the main and standby devices;

it should be noted that step 803 is an optional step, which may or may not be executed, and is not limited herein.

Steps 801 to 803 are similar to steps 701 to 703 in the embodiment corresponding to fig. 7, and are not repeated here.

804. When the execution of the data node fails, the main/standby switching is carried out;

when the data node executes the storage process, the coordinating node carries the recorded data with the intermediate state information of the storage process execution, and for a specific implementation process, reference is made to step 302 in the embodiment corresponding to fig. 3, which is not described herein again.

And when the execution of the data node storage process fails, reporting an error to the coordination node.

Optionally, there are multiple types of exceptions that cause the data node storage process to fail, and here, the specific reason for the storage process to fail is not limited.

If the main data node crashes and crashes, the working state of the standby data node is normal, the background cluster management module senses the situation and coordinates the node indication to carry out the main-standby switching. The switching of the main and standby nodes is usually controlled by a cluster management system, the cluster management system is mainly responsible for starting and stopping the whole cluster, collects the running state of each node, and controls the main and standby switching by sending a command to the data node when the running is abnormal.

805. The storage procedure is retried.

The coordinating node retries according to the recorded intermediate state information in the storage process, including sending an execution task to the data backup node for retrying, and for a specific implementation process, reference is made to step 303 in the embodiment corresponding to fig. 3, which is not described herein again.

Optionally, the coordinating node may synchronize data with other coordinating nodes in a synchronization manner, or store an intermediate variable of the execution result set before the execution failure of the storage process in a disk-dropping manner, so that retry of the execution failure of the storage process can be realized even if the coordinating node crashes.

Please refer to fig. 9, which is a schematic diagram of a system architecture of a retry method for failed execution of a storage process according to an embodiment of the present application.

The database management system stores the position of the executed transaction ending statement and the intermediate state information when the transaction ends in the execution of the storage process, and resends the storage process request to retry after capturing the abnormality. Because the data processing method of the embodiment records the position when the transaction is ended and the corresponding intermediate state information, fine-grained retry can be realized, and the customer experience when software and hardware are abnormal is improved.

The data processing method for implementing the retry of the execution failure of the storage procedure is described above, and the data processing apparatus for implementing the method is described below.

Please refer to fig. 10, which is a diagram illustrating an embodiment of a database management system according to an embodiment of the present application.

The database management system provided in the embodiment of the present application may be deployed in a stand-alone database system or nodes in various types of clustered database systems, where the nodes may run in a physical machine or a virtual machine, and the details of which are not limited herein.

In practical applications, the data processing device for deploying the database management system may be a database server, or a dedicated data processing device, and is not limited herein.

The database management system 1000 provided in the embodiment of the present application includes:

an execution unit 1001 for executing a stored procedure in response to a stored procedure request, the stored procedure comprising one or more ordered transactions;

a recording unit 1002, configured to record, when execution of a first transaction of the one or more ordered transactions ends, tag information of the first transaction;

a retry unit 1003, configured to retry the transaction after the first transaction in the storage process according to the tag information of the first transaction, based on that an exception occurs during the execution of the storage process and a condition for retrying the storage process is met.

The recording unit 1002 is further configured to: the request statement that called the stored procedure is recorded.

The marker information includes a line number of an end statement of the first transaction.

The apparatus further comprises:

a recording unit 1002, configured to record intermediate state information of the stored procedure, where the intermediate state information includes one or more of the following: the execution statement result of the first transaction, error prompt information, input parameters, output parameters, global variables, and temporary variables.

The intermediate state information includes an execution statement result or error prompt information of the first transaction, and the apparatus further includes:

the caching unit 1004 is configured to cache an execution statement result or error prompt information of the first transaction until the first transaction is ended, and then send the execution statement result or error prompt information to the user equipment.

the caching unit 1004 is configured to cache the execution statement result or the error prompt information of the first transaction until the storage process is finished, and then send the execution statement result or the error prompt information to the user equipment.

The first transaction is located at a branch of the branch statement in the storing process, and the recording unit 1002 is further configured to:

recording intermediate state information of the stored process, wherein the intermediate state information comprises a branch execution state, and the branch execution state is used for identifying a branch where the first transaction is located;

the retry unit 1003 is specifically configured to execute the transaction after the first transaction in the branch where the first transaction is located, based on the branch execution status.

The branch execution state includes a conditional result of executing the branch statement in the storing process.

The condition for retrying the stored procedure is that the stored procedure exception is a recoverability exception, the recoverability exception including: network exceptions, memory usage exceptions, input output exceptions, lock wait timeout exceptions, or two-phase transaction commit exceptions.

The retry unit 1003 is specifically configured to: and after waiting for the preset first time length, retrying the transaction after the first transaction in the storage process.

The recording unit 1002 is further configured to record, when execution of a second transaction of the one or more ordered transactions is finished, tag information of the second transaction, where the second transaction is executed after the first transaction;

the retry unit 1003 is specifically configured to: and retrying the transactions after the second transaction in the storage process according to the marking information of the second transaction, wherein the second transaction is the last executed transaction in the executed transactions when the condition of retrying the storage process is met.

Please refer to fig. 11, which is a diagram illustrating another exemplary database management system according to an embodiment of the present application.

The database management system can comprise a plurality of software functional modules, including a database server framework (DB frame), a database compiler, a database executor, a database exception handling module and the like, and the newly added sub-modules realize retry logic. Wherein:

1. the database server framework is used for analyzing the messages sent by the users to determine which messages are, such as SQL messages, PBE messages and the like. The system is also used for storing the original request statement and preparing for use in retry; but also for intermediate state information, etc.

2. A database compiler: and the method is used for limiting application scene check, and if a non-support scene is encountered, a no-retry mark is set.

3. A database executor: the method is used for storing and storing process variables and parameters, storing the position of the transaction end, including the positions of COMMIT and ROLLBACK, and storing the intermediate state information of the transaction end, including the first process parameters, the branch execution state, the execution result of the first variables and the like.

4. The storage process exception handling module: used for catching the exception, check whether to need retrying; but also for rollback transactions.

The database server may include: a re-execution module for starting a new transaction; skipping the committed statement according to the saved committed location information and the if branch judgment result when the last execution is carried out; and recovering the current state according to the information of the COMMIT point during the last execution, wherein the information comprises variables and parameters and the information of the result set during the COMMIT.

It should be noted that the precondition includes that data is not sent to the client before the storage process is executed; and the failure is a processable failure, the program internal failure retry is meaningless.

5. The abnormal retry module restarts a new transaction; skipping the submitted statement according to the saved COMMIT position information, if branch judgment result and other intermediate state information when the execution is carried out last time; and recovering the current state including variables, parameters and the like according to the intermediate state information corresponding to the COMMIT point during the last execution.

Please refer to fig. 12, which is a diagram illustrating an embodiment of a data processing apparatus according to an embodiment of the present application.

The data processing device provided in this embodiment may be a server or a dedicated data processing device, such as a database server, and the specific device form is not limited in this embodiment.

The data processing apparatus 1200 may have a large difference due to a difference in configuration or performance, and may include one or more processors 1201 and a memory 1202, where the memory 1202 stores programs or data.

Memory 1202 may be volatile memory or non-volatile memory, among others. Optionally, the processor 1201 is one or more Central Processing Units (CPUs) or a dedicated processor, and the CPU may be a single-core CPU or a multi-core CPU. The processor 1201 may communicate with the memory 1202 to execute a series of instructions in the memory 1202 on the data processing device 1200.

The data processing apparatus 1200 also includes one or more wired or wireless network interfaces 1203, such as an ethernet interface.

Optionally, although not shown in FIG. 12, the data processing apparatus 1200 may also include one or more power supplies; the input/output interface may be used to connect a display, a mouse, a keyboard, a touch screen device, a sensing device, or the like, and the input/output interface is an optional component, and may or may not be present, and is not limited herein.

The process executed by the processor 1201 in the data processing apparatus 1200 in this embodiment may refer to the method process described in the foregoing method embodiment, which is not described herein again.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A method of data processing, comprising:

in response to a stored procedure request, executing a stored procedure, the stored procedure comprising one or more ordered transactions;

when the execution of a first transaction in the one or more ordered transactions is finished, recording mark information of the first transaction;

and retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction based on the condition that the exception occurs in the storage process and the retrying of the storage process is met.

2. The method of claim 1, further comprising:

and recording a request statement for calling the storage process.

3. The method of claim 1 or 2, wherein the marker information comprises a line number of an end statement of the first transaction.

4. The method according to any one of claims 1 to 3, further comprising:

recording intermediate state information of the stored procedure, the intermediate state information including one or more of: the execution statement result of the first transaction, error prompt information, input parameters, output parameters, global variables and temporary variables.

5. The method of claim 4, wherein the intermediate state information comprises an execution statement result or error hint information for the first transaction, the method further comprising:

and caching the execution statement result or the error prompt message of the first transaction until the first transaction is ended, and then sending the execution statement result or the error prompt message to user equipment.

6. The method of claim 4, wherein the intermediate state information comprises an execution statement result or error hint information for the first transaction, the method further comprising:

and caching the execution statement result or the error prompt message of the first transaction until the storage process is finished, and then sending the execution statement result or the error prompt message to user equipment.

7. A method according to any one of claims 1 to 3, wherein the first transaction is located in a branch of a branch statement in the stored procedure, the method further comprising:

recording intermediate state information of the stored procedure, wherein the intermediate state information comprises a branch execution state, and the branch execution state is used for identifying a branch where the first transaction is located;

retrying transactions subsequent to the first transaction in the storage process comprises: based on the branch execution state, executing transactions subsequent to the first transaction in the branch in which the first transaction is located.

8. The method of claim 7,

the branch execution state comprises a conditional judgment result of executing the branch statement in the storage process.

9. The method of any of claims 1 to 8, wherein the condition for retrying the stored procedure is that the stored procedure exception is a recoverability exception, the recoverability exception comprising: network exceptions, memory usage exceptions, input output exceptions, lock wait timeout exceptions, or two-phase transaction commit exceptions.

10. The method of any of claims 1-9, wherein retrying transactions subsequent to the first transaction in the storage process comprises:

and after waiting for a preset first duration, retrying the transaction after the first transaction in the storage process.

11. The method according to any one of claims 1 to 10, further comprising:

when execution of a second transaction in the one or more ordered transactions is finished, recording mark information of the second transaction, wherein the second transaction is executed after the first transaction;

retrying transactions subsequent to the first transaction in the storage process comprises:

and retrying the transactions after the second transaction in the storage process according to the marking information of the second transaction, wherein the second transaction is the last executed transaction in the executed transactions when the condition of retrying the storage process is met.

12. A database management system, comprising:

an execution unit to execute a stored procedure in response to a stored procedure request, the stored procedure comprising one or more ordered transactions;

the recording unit is used for recording mark information of a first transaction in the one or more ordered transactions when the execution of the first transaction is finished;

and the retry unit is used for retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction based on the condition that the exception occurs in the storage process and the retrying of the storage process is met.

13. The database management system of claim 12, wherein the recording unit is further configured to:

and recording a request statement for calling the storage process.

14. A database management system according to claim 12 or 13, wherein said flag information includes a line number of an end statement of said first transaction.

15. A database management system as recited in any of claims 12 to 14 wherein the recording unit is further configured to:

16. The database management system of claim 15, wherein the intermediate state information comprises an execution statement result or error prompt information for the first transaction, the database management system further comprising:

and the cache unit is used for caching the execution statement result or the error prompt information of the first transaction until the first transaction is ended, and then sending the execution statement result or the error prompt information to user equipment.

17. The database management system of claim 15, wherein the intermediate state information comprises an execution statement result or error prompt information for the first transaction, the database management system further comprising:

and the cache unit is used for caching the execution statement result or the error prompt information of the first transaction until the storage process is finished, and then sending the execution statement result or the error prompt information to the user equipment.

18. Database management system according to any of claims 12 to 14, wherein said first transaction is located in a branch of a branch statement in said stored procedure, said recording unit further being adapted to:

the retry unit is specifically configured to execute, based on the branch execution state, a transaction subsequent to a first transaction in a branch in which the first transaction is located.

19. The database management system according to claim 18, wherein the branch execution state includes a conditional outcome of executing the branch statement in the stored procedure, the conditional outcome determining which branch of the branch statement to execute.

20. The database management system according to any of claims 12 to 19, wherein the condition for retrying the stored procedure is that the stored procedure exception is a recoverability exception, the recoverability exception comprising: network exceptions, memory usage exceptions, input output exceptions, lock wait timeout exceptions, or two-phase transaction commit exceptions.

21. Database management system according to any of claims 12 to 20, wherein the retry unit is specifically configured to:

22. Database management system according to any of claims 12 to 21,

the recording unit is further configured to record, when execution of a second transaction of the one or more ordered transactions is finished, tag information of the second transaction, where the second transaction is a transaction executed after the first transaction;

the retry unit is specifically configured to:

23. A data processing apparatus, characterized by comprising:

a memory to store instructions;

a processor for executing instructions in the memory to cause the data processing apparatus to perform the method of any of claims 1 to 12.

24. A computer program product, characterized in that it comprises instructions which, when run on a computer, cause the computer to carry out the method of any one of claims 1 to 12.

25. A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1 to 12.

26. A database system, comprising: the network interface is used for receiving a storage process request sent by user equipment; a data processing apparatus for performing the method of any of claims 1 to 12 in response to the stored procedure request.