CN113986923A

CN113986923A - Distributed transaction processing method, device and equipment based on only additional file storage

Info

Publication number: CN113986923A
Application number: CN202111262557.9A
Authority: CN
Inventors: 吕文栋; 洪毅清; 王一洲
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2021-10-28
Filing date: 2021-10-28
Publication date: 2022-01-28

Abstract

The present specification relates to the technical field of big data storage processing, and provides a distributed transaction processing method, device and equipment based on only additional file storage, wherein the method is applied to a master node side of a distributed database system, and comprises the following steps: receiving a data insertion request sent by a client; dispatching an append only operation task to a plurality of slave nodes in accordance with the data insertion request to execute the append only operation task on a swimlane basis in a distributed file system by the plurality of slave nodes; receiving the appending-only operation results returned by the plurality of slave nodes; and submitting the result of the append-only operation returned by the plurality of slave nodes as a transaction. The embodiment of the specification can improve the data consistency and the processing efficiency of the distributed transaction processing.

Description

Distributed transaction processing method, device and equipment based on only additional file storage

Technical Field

The present specification relates to the field of big data storage technologies, and in particular, to a distributed transaction processing method, apparatus, and device based on append-only file storage.

Background

In a distributed scenario, distributed transactions are typically implemented according to the XA protocol, a standard for interfacing resource managers (databases) with transaction managers. However, there is a possibility of data inconsistency in this way. For example, suppose the coordinator issues a notification of a commit (commit) of a transaction, but the notification is received by only a portion of the transaction participants and commit operations are performed because of network problems, etc., and the rest of the transaction participants are always in a blocked state because no notification is received.

Disclosure of Invention

An object of the embodiments of the present specification is to provide a distributed transaction processing method, apparatus, and device based on only additional file storage, so as to improve data consistency and processing efficiency of distributed transaction processing.

In order to achieve the above object, in one aspect, an embodiment of the present specification provides a distributed transaction processing method based on append-only file storage, which is applied to a master node side of a distributed database system, where the method includes:

receiving a data insertion request sent by a client;

dispatching an append only operation task to a plurality of slave nodes in accordance with the data insertion request to execute the append only operation task on a swimlane basis in a distributed file system by the plurality of slave nodes;

receiving the appending-only operation results returned by the plurality of slave nodes;

and submitting the result of the append-only operation returned by the plurality of slave nodes as a transaction.

In this embodiment of this specification, the committing the append-only operation results returned by the plurality of slave nodes as one transaction includes:

judging whether the only adding operations of the plurality of slave nodes are successful or not according to the only adding operation results returned by the plurality of slave nodes;

when the append-only operations of the plurality of slave nodes are all successful, updating local metadata;

dispatching a rollback operation task to the plurality of slave nodes to execute the rollback operation task by the plurality of slave nodes on a swim lane basis in the distributed file system when the append-only operation of the slave nodes is unsuccessful.

In the embodiment of the present specification, an append-only table is provided in the metadata; the append only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field, and a file pre-compression logical length field.

In the embodiment of the present specification, a columnar storage table is provided in the metadata; the columnar storage table comprises a slave node field, a file actual logical length field, a file tuple number field and a file pre-compression logical length field.

In this embodiment of the present specification, each of the lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

In an embodiment of this specification, the distributed database system includes an MPP distributed database system.

In an embodiment of the present description, the distributed file system includes a Hadoop distributed file system.

On the other hand, the embodiments of the present specification further provide a distributed transaction processing apparatus based on append-only file storage, including:

the request receiving module is used for receiving a data insertion request sent by a client;

a task dispatching module for dispatching append-only operation tasks to a plurality of slave nodes according to the data insertion requests, so that the append-only operation tasks are executed by the plurality of slave nodes on a swimlane basis in a distributed file system;

the result receiving module is used for receiving the result of the only appending operation returned by the plurality of slave nodes;

and the transaction committing module is used for committing the result of the only appending operation returned by the plurality of slave nodes as a transaction.

In another aspect, the embodiments of the present specification further provide a computer device, which includes a memory, a processor, and a computer program stored on the memory, and when the computer program is executed by the processor, the computer program executes the instructions of the above method.

In another aspect, the present specification further provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor of a computer device to execute the instructions of the method.

In another aspect, the present specification further provides a computer program product, which when executed by a processor of a computer device, executes the instructions of the method described above.

On the other hand, the embodiment of the present specification further provides another distributed transaction processing method based on append-only file storage, which is applied to a slave node side of a distributed database system, and the method includes:

receiving an additional operation task sent by a main node;

executing the append-only operation task based on the swim lane in the distributed file system to obtain an append-only operation result;

returning the result of the append-only operation to the master node, and submitting the append-only operation result returned by the slave nodes as a transaction by the master node.

In this embodiment of the present specification, the performing the append-only operation task based on a swimlane in a distributed file system includes:

determining a swim lane corresponding to the append-only operation task in the distributed file system;

opening a data file corresponding to the lane;

inserting a data block of the specified additional data in the additional-only operation task at the end of the content of the data file corresponding to the lane;

and closing the data file corresponding to the swim lane to prevent the handle from being leaked.

In this embodiment of this specification, the committing, by the master node, the append-only operation result returned by the plurality of slave nodes as one transaction includes:

judging whether the only-adding operations of the plurality of slave nodes are successful or not by the master node according to the only-adding operation results returned by the plurality of slave nodes;

when the appending-only operations of the plurality of slave nodes are successful, updating local metadata by the master node;

dispatching, by the master node, a rollback operation task to the plurality of slave nodes when the append-only operation of the slave nodes is unsuccessful.

In an embodiment of this specification, the method further includes:

when a rollback operation task is received, determining a swim lane corresponding to the rollback operation task in the distributed file system;

opening a data file corresponding to the lane;

clearing the tail data block of the data file corresponding to the lane;

On the other hand, the embodiments of the present specification further provide another distributed transaction processing apparatus based on append-only file storage, where the apparatus includes:

the task receiving module is used for receiving the additional operation task sent by the main node;

the task execution module is used for executing the task of the only additional operation based on the swim lane in the distributed file system to obtain the result of the only additional operation;

and the result returning module is used for returning the result of the only adding operation to the main node so that the main node submits the result of the only adding operation returned by the plurality of slave nodes as a transaction.

In another aspect, the present specification provides another computer device, including a memory, a processor, and a computer program stored in the memory, where the computer program is executed by the processor to execute the instructions of the method.

On the other hand, the embodiments of the present specification also provide another computer storage medium, on which a computer program is stored, and the computer program is executed by a processor of a computer device to execute the instructions of the method.

In another aspect, the embodiments of the present specification also provide another computer program product, which when executed by a processor of a computer device, executes the instructions of the above method.

As can be seen from the technical solutions provided by the embodiments of the present specification, in the embodiments of the present specification, since the transaction commit is performed only on the master node, the slave nodes do not participate in the transaction commit, and the master node commits the only-appending operation result returned by the plurality of slave nodes as one transaction, either successfully or unsuccessfully, and the situations that a part of the transactions are successfully committed and a part of the transactions are unsuccessfully committed, as in the prior art, are not caused, so that the data consistency of the distributed transaction processing is improved. In addition, the slave node executes the task of the only adding operation based on the swim lane in the distributed file system, and the data concurrent only adding operation is realized, so that the processing efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort. In the drawings:

FIG. 1 illustrates a diagram of a prior art distributed transaction two-stage commit;

FIG. 2 illustrates a block diagram of a distributed transaction processing system based on append-only file storage in some embodiments of the present description;

FIG. 3 illustrates a flow diagram of a distributed transaction processing method (primary node side) based on append-only file storage according to some embodiments of the present description;

FIG. 4 is a schematic diagram of a lane model in one embodiment of the present specification;

FIG. 5 is a diagram illustrating a storage structure of an append only table in an embodiment of the present specification;

FIG. 6 is a diagram illustrating a storage structure of a columnar storage table in an embodiment of the present specification;

FIG. 7 is a flow diagram illustrating a distributed transaction processing method (from the node side) based on append-only file storage in accordance with further embodiments of the present description;

FIG. 8 illustrates an append file only storage based distributed transaction processing flow diagram for an append file only storage based distributed transaction processing system of some embodiments of the present description;

FIG. 9 illustrates a block diagram of a distributed transaction processing apparatus based on append-only file storage in some embodiments of the present description;

FIG. 10 is a block diagram of a distributed transaction processing apparatus based on append-only file storage in further embodiments of the present description;

FIG. 11 is a block diagram illustrating the architecture of a computer device in some embodiments of the present description.

[ description of reference ]

10. A client;

20. a distributed database system;

30. a distributed file system;

91. a request receiving module;

92. a task dispatching module;

93. a result receiving module;

94. a transaction commit module;

11. a task receiving module;

12. a task execution module;

13. a result returning module;

1102. a computer device;

1104. a processor;

1106. a memory;

1108. a drive mechanism;

1110. an input/output interface;

1112. an input device;

1114. an output device;

1116. a presentation device;

1118. a graphical user interface;

1120. a network interface;

1122. a communication link;

1124. a communication bus.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step should fall within the scope of protection of the present specification.

The embodiment of the present specification relates to an Append-only operation; the addition is an operation mode in which all the contents to be added at this time are written at the end of the contents of the original file every time the addition is performed, and is essentially a special writing mode. If the appended file does not exist, a file of the specified name is created at the specified location. Accordingly, a file on which an append operation is performed (i.e., a data file of the distributed file system in the embodiment of the present specification) may also be referred to as an append-only file.

As shown in FIG. 1, the prior art XA protocol-based distributed transaction is divided into two phases, a first phase: the transaction manager requires a separate pre-commit (commit) operation for each resource manager involved in the transaction and reflects whether it can commit. And a second stage: the transaction manager requires each resource manager to either commit (commit) the data individually, or to rollback the data individually. However, if the transaction manager issues a notification of a transaction commit (commit); however, because of network problems, the notification is received by only a part of the resource managers and commit operation is performed, and the rest of the resource managers are in a blocking state because the notification is not received, and the resources are not released until the commit is completed, so that the problem of data inconsistency is generated, and the blocking state affects the efficiency of the distributed transaction processing.

In view of the above, the embodiments of the present specification provide a distributed transaction processing system based on append-only file storage. The distributed transaction processing system based on the append-only file storage can be applied to any distributed system with similar problems, such as a bank system, a telecommunication service system, an e-commerce platform and the like. Referring to FIG. 2, in some embodiments, the append only file storage based distributed transaction system may include a client 10, a distributed database system 20, and a distributed file system 30. The client 10 is used to initiate a data insertion request to the distributed database system 20. The distributed database system 20 may include a master node and a plurality of slave nodes. The main node is used for receiving a data insertion request sent by a client; dispatching the append only operation task to the plurality of slave nodes based on the data insertion request to execute the append only operation task on a swimlane basis in the distributed file system 30 by the plurality of slave nodes; receiving the appending-only operation results returned by a plurality of slave nodes; and submitting the result of the append-only operation returned by a plurality of slave nodes as a transaction.

The transaction submission is only carried out on the master node, the slave nodes do not participate in the transaction submission, and the master node submits the only-additional operation result returned by the plurality of slave nodes as one transaction, wherein the result is either successful or failed, and the situations that a part of transactions are successfully submitted and a part of transactions are unsuccessfully submitted in the prior art are avoided, so that the data consistency of the distributed transaction processing is improved. In addition, the slave node executes the task of the only adding operation based on the swim lane in the distributed file system, and the data concurrent only adding operation is realized, so that the processing efficiency is improved.

In some embodiments, the distributed database system may be an electronic device with computing and network interaction functions; software that runs in the electronic device and provides business logic for data processing and network interaction is also possible. The user side can be self-service terminal equipment, a mobile terminal (namely a smart phone), a display, a desktop computer, a tablet computer, a notebook computer, a digital assistant or intelligent wearable equipment and the like. Wherein, wearable equipment of intelligence can include intelligent bracelet, intelligent wrist-watch, intelligent glasses or intelligent helmet etc.. Of course, the user end is not limited to the electronic device with a certain entity, and may also be software running in the electronic device. The distributed file system can be an electronic device with operation and network interaction functions; software that runs in the electronic device and provides business logic for data processing and network interaction is also possible. The distributed file system of the embodiment of the specification supports only an append operation and supports a flush operation (Truncate) of a data Block (Block); the emptying operation of the data block is used for emptying the data block at the tail end of the specified file.

In some embodiments, the Distributed database System may be an MPP (Massive Parallel Processing) Distributed database System, and the Distributed File System may be an HDFS (Hadoop Distributed File System) File System. Therefore, by utilizing the characteristic that the HDFS file system can only be added (update and delete operations cannot be carried out) and the MPP distributed transaction processing mode, the transaction processing model can be simplified, and the efficient writing of data is ensured.

In the embodiment of the present specification, the Append-only operation is an operation mode in which all contents to be appended at this time are written at the end of the original file contents every time of appending, and is essentially a special writing mode. If the appended file does not exist, a file of the specified name is created at the specified location. Accordingly, a file on which an append operation is performed (i.e., a data file of the distributed file system in the embodiment of the present specification) may also be referred to as an append-only file.

Referring to fig. 8, when data needs to be inserted, a user may input an insertion statement through a client so as to initiate a data insertion request to a master node of the distributed database system through the client. The data insertion request may carry one or more pieces of user data to be inserted. For example, in an exemplary embodiment, the user data to be inserted may include: the user XX deposits 1 ten thousand yuan into the account ZZ through the self-service terminal YY at 12:00:00 of 9, 1 and 1 month in 2021; the account MM transfers 1 ten-thousand yuan to the account NN through a mobile phone bank APP at 12:00:00 of 9, 1 and 1 month in 2021; and so on. When the master node receives the data insertion request, one or more (generally, a plurality of) slave nodes can be selected from the slave node set of the distributed database system according to the data insertion request, and the slave nodes are dispatched to the append-only operation task, so that the slave nodes can execute the append-only operation task based on a swim lane in the distributed file system, and the user data can be appended to the data file of the distributed file system. For example, in an exemplary embodiment, selected ones of the slave nodes are assigned inserts for storing business data, ones of the slave nodes are assigned inserts for transferring business data, and yet others of the slave nodes are assigned inserts for retrieving business data, among other things.

To improve the efficiency of distributed transactions, a swim lane model is used in the embodiments of the present specification. The swim lane model divides each swim lane according to roles (for example, only adding processes), organizes activities (for example, only adding data operations) according to duties, and scatters the activities of each role in the swim lane corresponding to each role. As such, different activities of different roles are divided into different regions (i.e., logically and/or physically isolated) referred to as swimlanes. The lanes are not called each other; the activity in any lane is abnormal, and other lanes are not affected, so that error isolation can be realized.

The swim lane model can be configured in advance in the distributed file system, the number of the swim lanes in the swim lane model can be determined according to the number of slave nodes in the slave node set and the upper limit of concurrent transactions, and each swim lane corresponds to one data file (namely a user data file). For example, in an exemplary embodiment, the number of slave nodes is 100, and the concurrent transaction upper limit is 20, then the upper limit of the number of lanes of the pre-configured lane model is 2000 — 100 × 20. In real time, the corresponding lanes may be formed as needed. Wherein, each lane may include a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append only operation task identifier.

In some embodiments, assuming that the distributed database system has N slave nodes (segments), for a single insertion operation of user data, each slave node only needs to write data into a separate file, in other words, N data files are simultaneously opened and written with data in an append-only manner. For the concurrent insertion operation of user data, different concurrent operations are written into different data files without mutual interference, and when there are C concurrent insertion operations, the number of files which are simultaneously opened and written is equal to N multiplied by C.

For example, in an exemplary embodiment, for a distributed database system with 3 slave nodes, the layout of the files on the corresponding swim lanes and distributed file system when performing 2 concurrent insertions may be as shown in FIG. 4. In fig. 4, the first transaction is insert 1, the second transaction is insert 2, and each insert has a file on each slave node for concurrent writing of data. There are 6 lanes formed, and the writing operation of each lane is performed independently without any influence. The specific execution steps for each lane are also the same as for a single insert operation. At this time, the paths of the 6 lanes on the distributed file system are respectively:

/filespace_oid/database_oid/table_oid/1，

/filespace_oid/database_oid/table_oid/2，

/filespace_oid/database_oid/table_oid/3，

/filespace_oid/database_oid/table_oid/4，

/filespace_oid/database_oid/table_oid/5，

/filespace_oid/database_oid/table_oid/6。

where filespace _ oid is the number of the file space, database _ oid is the number of the database, and table _ oid is the number of the table.

In some embodiments, for each slave node (i.e., each slave node dispatched by the master node with an append only operation task), the slave node performing the append only operation task on a swim lane basis in the distributed file system may include:

(1) and receiving an additional operation task sent by the main node.

(2) Executing the append-only operation task based on the swim lane in the distributed file system to obtain an append-only operation result, which may specifically include:

1) determining a lane corresponding to the only-additional-operation task in the distributed file system, wherein each only-additional-operation task corresponds to a slave node identifier and content to be added, and each lane can comprise a slave node identifier, a file space identifier, a database identifier, a user data table identifier and an only-additional-operation task identifier; accordingly, the slave node can find the corresponding lane to execute the append-only operation task in the lane.

2) And opening the data file corresponding to the lane.

3) And inserting a data block for specifying the additional data in the additional-only operation task at the end of the content of the data file corresponding to the lane. For example, in the data file corresponding to the lane before the current addition, five data blocks are sequentially arranged from the first to the second: data block A, data block B, data block C, data block D and data block E; then a new data block F can be added after the data block E in the current addition; that is, after the addition, six data blocks are sequentially arranged from the first to the second in the data file corresponding to the lane: data block a, data block B, data block C, data block D, data block E, and data block F.

4) And closing the data file corresponding to the swim lane to prevent the handle from being leaked.

(2) And returning the result of the only-appending operation to the master node, so that the master node commits the result of the only-appending operation returned by the slave nodes as a transaction.

In the embodiment of the present specification, a complete data addition includes not only an addition-only operation (i.e. writing user data) on a data file in the distributed file system, but also an update operation on metadata locally stored in the distributed database. The metadata is data used to describe the data file. Therefore, each slave node needs to return the append-only operation result to the master node, so that the master node commits the append-only operation result returned by each slave node as a transaction to ensure data consistency.

In some embodiments, for a master node (e.g., a transaction manager of the master node), committing the append-only operation results returned by multiple slave nodes as one transaction may include:

(1) and judging whether the only adding operations of the plurality of slave nodes are successful or not according to the only adding operation results returned by the plurality of slave nodes.

(2) And when the appending-only operations of the plurality of slave nodes are successful, updating the local metadata.

When the append-only operations of the plurality of slave nodes are successful, the append-only operations for the data insertion request are successful, and therefore the local metadata can be updated, so that the metadata for recording the file information and the data file on the append-only distributed file storage are corresponded, and the whole data insertion transaction for the data insertion request is completed.

In this embodiment, for each user data table, a system table may be added to the metadata (Catalog) and named by the following rules: for the AO table (i.e., the append only table), use pg _ aoseg. pg _ aoseg _ $ tableoil; for the partial table (i.e., the columnar storage table), pg _ aoseg. pg _ paqseg _ $ tableoil may be used. Referring to fig. 5, the AO table may include a slave node field (segno), a file actual logical length field (eof), a file tuple number field (tupcount), a file variable length block number field (varlockcount), and a file pre-compression logical length field (eofung compressed). Referring to fig. 6, the partial table may include a slave node field (segno), a file actual logical length field (eof), a file tuple number field (tupcount), and a file pre-compression logical length field (eoffrecompressed). When a data file in the distributed file system is added with data, the actual logical length of the corresponding file, the number of file tuples, the number of file variable-length blocks, the logical length before file compression, and the like will change, so when updating metadata, the fields related to the AO table or the queue table need to be modified to maintain data consistency.

(3) And dispatching a rollback operation task to the plurality of slave nodes when the append-only operation of the slave node is unsuccessful, so that the rollback operation task is executed by the plurality of slave nodes on a swim lane basis in the distributed file system.

If the append-only operation of any slave node is unsuccessful, indicating that the append-only operation for the data insertion request fails as a whole, the local metadata cannot be updated (i.e., the local metadata is maintained), and therefore, roll-back operation tasks need to be dispatched to the plurality of slave nodes to maintain data consistency.

In some embodiments, for each slave node, performing the rollback operation task may include:

(1) and when a rollback operation task is received, determining a corresponding lane of the rollback operation task in the distributed file system.

(2) And opening the data file corresponding to the lane.

(3) And emptying the tail data block of the data file corresponding to the lane. For example, in an exemplary embodiment, after the appending, the data file corresponding to the swimlane has six data blocks from the first to the second in sequence: a data block A, a data block B, a data block C, a data block D, a data block E and a data block F; if the data block F is a newly added data block, performing a rollback operation is to perform a clear (round) operation on the data block F.

(4) And closing the data file corresponding to the swim lane to prevent the handle from being leaked.

Referring to fig. 8, in this embodiment of the present specification, after confirming that the transaction is committed or rolled back, the master node may further return a data insertion response to the client to feed back the transaction processing result to the user.

Referring to fig. 3, an embodiment of the present specification provides a distributed transaction processing method based on append-only file storage, which may be applied to a master node side of a distributed database system, where the method may include the following steps:

s301, receiving a data insertion request sent by a client.

S302, dispatching an append only operation task to a plurality of slave nodes according to the data insertion request, and executing the append only operation task by the slave nodes on a swimlane basis in a distributed file system.

S303, receiving the result of the only adding operation returned by the plurality of slave nodes.

S304, submitting the result of the only appending operation returned by the plurality of slave nodes as a transaction.

In the embodiment of the description, the transaction submission is only performed on the master node, the slave nodes do not participate in the transaction submission, and the master node submits the only-additional operation result returned by the plurality of slave nodes as one transaction, either successfully or unsuccessfully, so that the situations that a part of transactions are successfully submitted and a part of transactions are unsuccessfully submitted in the prior art are avoided, and the data consistency of the distributed transaction processing is improved. In addition, the slave node executes the task of the only adding operation based on the swim lane in the distributed file system, and the data concurrent only adding operation is realized, so that the processing efficiency is improved.

In the embodiment shown in fig. 3, committing the append-only operation results returned by the plurality of slave nodes as a transaction may include:

In the embodiment shown in fig. 3, an append-only table is set in the metadata; the append only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field, and a file pre-compression logical length field.

In the embodiment shown in fig. 3, a columnar storage table is arranged in the metadata; the columnar storage table comprises a slave node field, a file actual logical length field, a file tuple number field and a file pre-compression logical length field.

In the embodiment shown in FIG. 3, each of the swimlanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append only operation task identifier.

In the embodiment shown in FIG. 3, the distributed database system comprises a MPP distributed database system.

In the embodiment shown in FIG. 3, the distributed file system comprises a Hadoop distributed file system.

Referring to fig. 7, an embodiment of the present specification provides another distributed transaction processing method based on append-only file storage, which may be applied to a slave node side of a distributed database system, and the method may include the following steps:

s701, receiving an additional operation task sent by the main node.

S702, executing the task of the only adding operation based on the swim lane in the distributed file system, and obtaining the result of the only adding operation.

S703, returning the result of the only appending operation to the master node, and submitting the result of the only appending operation returned by the slave nodes as a transaction by the master node.

In the embodiment shown in fig. 7, the performing the append only operation task based on a swimlane in a distributed file system includes:

opening a data file corresponding to the lane;

In the embodiment shown in fig. 7, the committing, by the master node, the append-only operation result returned by the plurality of slave nodes as one transaction includes:

In the embodiment shown in fig. 7, the method further comprises:

opening a data file corresponding to the lane;

clearing the tail data block of the data file corresponding to the lane;

In the embodiment shown in fig. 7, an append-only table is provided in the metadata; the append only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field, and a file pre-compression logical length field.

In the embodiment shown in fig. 7, a columnar storage table is provided in the metadata; the columnar storage table comprises a slave node field, a file actual logical length field, a file tuple number field and a file pre-compression logical length field.

In the embodiment shown in FIG. 7, each of the swimlanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append only operation task identifier.

In the embodiment shown in FIG. 7, the distributed database system comprises a MPP distributed database system.

In the embodiment shown in FIG. 7, the distributed file system comprises a Hadoop distributed file system. While the process flows described above include operations that occur in a particular order, it should be appreciated that the processes may include more or less operations that are performed sequentially or in parallel (e.g., using parallel processors or a multi-threaded environment).

Corresponding to the distributed transaction processing method based on only additional file storage shown in fig. 3, an embodiment of the present specification further provides a distributed transaction processing apparatus based on only additional file storage, and as shown in fig. 9, the distributed transaction processing apparatus based on only additional file storage may include: a request receiving module 91, a task dispatching module 92, a result receiving module 93, and a transaction committing module 94. Wherein:

the request receiving module 91 may be configured to receive a data insertion request sent by a client.

A task dispatching module 92, which may be configured to dispatch an append only operation task to a plurality of slave nodes according to the data insertion request, so that the append only operation task is executed by the plurality of slave nodes on a swimlane basis in the distributed file system.

The result receiving module 93 may be configured to receive the append-only operation result returned by the plurality of slave nodes.

The transaction commit module 94 may be configured to commit the append-only operation results returned by the plurality of slave nodes as a transaction.

In correspondence to the distributed transaction processing method based on only append file storage shown in fig. 7, an embodiment of the present specification further provides another distributed transaction processing apparatus based on only append file storage, and as shown in fig. 10, the distributed transaction processing apparatus based on only append file storage may include: a task receiving module 11, a task executing module 12 and a result returning module 13. Wherein:

the task receiving module 11 may be configured to receive an additional operation task sent by the host node.

The task execution module 12 may be configured to execute the append-only operation task based on a swimlane in the distributed file system, and obtain an append-only operation result.

The result returning module 13 may be configured to return the append-only operation result to the master node, so that the master node commits the append-only operation results returned by multiple slave nodes as one transaction.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

Embodiments of the present description also provide a computer device. As shown in FIG. 11, in some embodiments of the present description, the computer device 1102 may include one or more processors 1104, such as one or more Central Processing Units (CPUs) or Graphics Processors (GPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106 for storing any kind of information such as code, settings, data, etc., and in one embodiment, a computer program stored on the memory 1106 and executable on the processor 1104, the computer program when executed by the processor 1104 may perform the instructions of the append only file storage based distributed transaction method according to any of the embodiments described above. For example, and without limitation, memory 1106 may include any one or more of the following in combination: any type of RAM, any type of ROM, flash memory devices, hard disks, optical disks, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent fixed or removable components of computer device 1102. In one case, when the processor 1104 executes the associated instructions, which are stored in any memory or combination of memories, the computer device 1102 can perform any of the operations of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108, such as a hard disk drive mechanism, an optical disk drive mechanism, etc., for interacting with any memory.

Computer device 1102 can also include input/output interface 1110(I/O) for receiving various inputs (via input device 1112) and for providing various outputs (via output device 1114). One particular output mechanism may include a presentation device 1116 and an associated graphical user interface 1118 (GUI). In other embodiments, input/output interfaces 1110(I/O), input device 1112, and output device 1114 may also be excluded, as only one computer device in a network. Computer device 1102 can also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the above-described components together.

Communication link 1122 may be implemented in any manner, e.g., via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. Communications link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products of some embodiments of the specification. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processor to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computer device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The described embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

It should also be understood that, in the embodiment of the present specification, the term "and/or" is only one kind of association relation describing an associated object, and means that three kinds of relations may exist. For example, a and/or B, may represent: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the method and apparatus embodiments, since they are substantially similar to the system embodiments, the description is relatively simple, and reference may be made to some descriptions of the system embodiments for relevant points.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A distributed transaction processing method based on only additional file storage is applied to a main node side of a distributed database system, and is characterized in that the method comprises the following steps:

receiving a data insertion request sent by a client;

2. The append-only file storage based distributed transaction processing method of claim 1, wherein committing the append-only operation results returned by the plurality of slave nodes as a transaction comprises:

3. The append only file storage based distributed transaction method of claim 2, wherein an append only table is provided in the metadata; the append only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field, and a file pre-compression logical length field.

4. The append only file storage based distributed transaction method of claim 2 wherein a columnar storage table is provided in the metadata; the columnar storage table comprises a slave node field, a file actual logical length field, a file tuple number field and a file pre-compression logical length field.

5. The append only file storage based distributed transaction method of claim 1, wherein each of the swimlanes comprises a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append only operation task identifier.

6. The append only file storage based distributed transaction method of claim 1, wherein the distributed database system comprises an MPP distributed database system.

7. The append only file storage based distributed transaction method of claim 1, wherein the distributed file system comprises a Hadoop distributed file system.

8. A distributed transaction processing apparatus based on append-only file storage, comprising:

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, performs the instructions of the method of any one of claims 1-7.

10. A computer storage medium on which a computer program is stored, characterized in that the computer program, when being executed by a processor of a computer device, executes instructions of a method according to any one of claims 1-7.

11. A computer program product, characterized in that the computer program product, when executed by a processor of a computer device, executes instructions of a method according to any one of claims 1-7.

12. A distributed transaction processing method based on append-only file storage is applied to a slave node side of a distributed database system, and is characterized in that the method comprises the following steps:

receiving an additional operation task sent by a main node;

13. The append only file storage based distributed transaction processing method of claim 12, wherein performing the append only operation task based on a swimlane in a distributed file system comprises:

opening a data file corresponding to the lane;

14. The append-only file storage based distributed transaction processing method of claim 13, wherein the committing, by the master node, the append-only operation results returned by the plurality of slave nodes as one transaction comprises:

15. The append only file storage based distributed transaction method of claim 14, wherein the method further comprises:

opening a data file corresponding to the lane;

clearing the tail data block of the data file corresponding to the lane;

16. The append only file storage based distributed transaction processing method of claim 14, wherein an append only table is provided in the metadata; the append only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field, and a file pre-compression logical length field.

17. The append only file storage based distributed transaction method of claim 14 wherein a columnar storage table is provided in the metadata; the columnar storage table comprises a slave node field, a file actual logical length field, a file tuple number field and a file pre-compression logical length field.

18. The append only file storage based distributed transaction method of claim 12 wherein each of the swimlanes includes a slave node identification, a file space identification, a database identification, a user data table identification, and an append only operation task identification.

19. The append only file storage based distributed transaction method of claim 12, wherein the distributed database system comprises an MPP distributed database system.

20. The append only file storage based distributed transaction method of claim 12, wherein the distributed file system comprises a Hadoop distributed file system.

21. A distributed transaction processing apparatus based on append-only file storage, comprising:

22. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, performs the instructions of the method of any one of claims 12-20.

23. A computer storage medium on which a computer program is stored, characterized in that the computer program, when being executed by a processor of a computer device, executes instructions of a method according to any one of claims 12-20.

24. A computer program product, characterized in that the computer program product, when executed by a processor of a computer device, executes instructions of a method according to any one of claims 12-20.