CN115185966A

CN115185966A - Method and device for processing data consistency in distributed cluster

Info

Publication number: CN115185966A
Application number: CN202210788586.7A
Authority: CN
Inventors: 周力
Original assignee: Jingdong Technology Information Technology Co Ltd
Current assignee: Jingdong Technology Information Technology Co Ltd
Priority date: 2022-07-06
Filing date: 2022-07-06
Publication date: 2022-10-14

Abstract

The invention discloses a method and a device for processing data consistency in a distributed cluster, and relates to the technical field of computers. One embodiment of the method comprises: acquiring a data processing request; the data processing request comprises a to-be-processed data identifier; determining a data table to be processed, one or more data files corresponding to the data table to be processed and the transaction information of each data file according to the data identifier to be processed; one or more data files are generated from one or more transactional operations; determining a query version number corresponding to a data table to be processed; determining a target data file corresponding to the data processing request according to the transaction information and the query version number; and processing the data corresponding to the to-be-processed data identification in the target data file. The implementation method avoids the situation that numerical values are different when a plurality of distributed nodes read the same data in distributed query, and ensures the data consistency of the distributed query.

Description

Method and device for processing data consistency in distributed cluster

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing data consistency in a distributed cluster.

Background

When data in a cloud-native distributed database is queried, a manner of querying the data by using a plurality of distributed nodes is generally adopted.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

when the data tables of the distributed database are continuously updated and a plurality of distributed nodes respectively query the same data table, the numerical values of the same data read may be different, so that the data are inconsistent.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for processing data consistency in a distributed cluster, where when any node in the distributed cluster is processed according to a same data processing request, a same target data file may be determined according to a query version number and transaction information of the data file; therefore, when distributed query is carried out, all the nodes can carry out query according to the same target data file, so that the condition that the nodes read the same numerical value of the same data is ensured, and the data consistency of the distributed query is further ensured.

Furthermore, the data table is stored by using the merging tree storage structure, a new data file is generated every time of data writing operations such as adding, updating or deleting data, and the data in the new data file is written, so that data files of multiple versions are obtained, the influence of the writing operations on the reading operations can be avoided, the reading and writing concurrency of the distributed database is improved, and the performance of the distributed database is further improved; in addition, a plurality of data files are combined regularly, so that the condition that a large number of small files appear is avoided, the storage space is saved, and the reading and writing efficiency is improved.

In order to achieve the above object, according to a first aspect of the embodiments of the present invention, a method for processing data consistency in a distributed cluster is provided, including:

acquiring a data processing request; the data processing request comprises a to-be-processed data identifier;

determining a data table to be processed, one or more data files corresponding to the data table to be processed and the transaction information of each data file according to the data identifier to be processed; the one or more data files are generated according to one or more transaction operations;

determining a query version number corresponding to the data table to be processed;

determining a target data file corresponding to the data processing request according to the transaction information and the query version number;

and processing the data corresponding to the to-be-processed data identifier in the target data file.

Optionally, the determining, according to the transaction information and the query version number, a target data file corresponding to the data processing request includes:

determining latest transaction information from the transaction information corresponding to one or more data files; taking the latest transaction information as the current transaction information, and circularly executing the following steps until the target data file is determined:

determining whether the query version number matches the current transaction information;

if so, taking the data file corresponding to the current transaction information as the target data file;

and if not, taking the previous transaction information of the current transaction information as the current transaction information.

Optionally, the determining whether the query version number matches the current transaction information includes:

and under the condition that the first transaction number is the same as the query version number, determining that the query version number is matched with the current transaction information.

Optionally, the current transaction information further includes a second transaction number, where the second transaction number corresponds to a transaction operation for modifying the data file; the determining whether the query version number matches the current transaction information includes:

in the case where the first transaction number is not greater than the query version number,

judging whether the second transaction number is larger than the query version number; if yes, determining that the query version number is matched with the current transaction information; if not, determining that the query version number is not matched with the current transaction information.

Optionally, the determining a query version number corresponding to the to-be-processed data table includes:

determining whether a transaction number corresponding to the data table to be processed exists in a cache; if yes, the transaction number is used as the query version number; if not, extracting the latest transaction number from the transaction manager, and taking the latest transaction number as the query version number.

Optionally, the to-be-processed data table uses a merged tree storage structure for data storage, so as to obtain a new data file after each transaction operation.

Optionally, before the processing the data corresponding to the to-be-processed data identifier in the target data file, the method includes:

determining that each node in the distributed cluster has synchronized the target data file.

Optionally, the processing the data corresponding to the to-be-processed data identifier in the target data file includes:

and inquiring, updating, deleting or adding new data corresponding to the to-be-processed data identification, or merging the target data file.

According to a second aspect of the embodiments of the present invention, there is provided a device for processing data consistency in a distributed cluster, including: the device comprises an acquisition module, a determination module and a processing module; wherein the content of the first and second substances,

the acquisition module is used for acquiring a data processing request; the data processing request indicates a to-be-processed data identifier;

the determining module is used for determining a data table to be processed, one or more data files corresponding to the data table to be processed and the transaction information of each data file according to the data identifier to be processed; the one or more data files are generated according to one or more transaction operations; determining a query version number corresponding to the data table to be processed; determining a target data file corresponding to the data processing request according to the transaction information and the query version number;

and the processing module is used for processing the data corresponding to the to-be-processed data identifier in the target data file.

According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods for processing data consistency in a distributed cluster as provided in the first aspect.

According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements any one of the methods for processing data consistency in a distributed cluster as provided in the first aspect above.

One embodiment of the above invention has the following advantages or benefits: when any node in the distributed cluster is processed according to the same data processing request, the same target data file can be determined according to the query version number and the transaction information of the data file; therefore, when distributed query is carried out, all the nodes can carry out query according to the same target data file, so that the condition that the nodes read the same numerical value of the same data is ensured, and the data consistency of the distributed query is further ensured.

Furthermore, the data table is stored by using the merged tree storage structure, a new data file is generated every time of data writing operations such as adding, updating or deleting data, and the data in the new data file is written, so that data files of multiple versions are obtained, the write operation can be prevented from influencing the read operation, the read-write concurrency of the distributed database is improved, and the performance of the distributed database is further improved; in addition, a plurality of data files are combined regularly, so that the condition that a large number of small files appear is avoided, the storage space is saved, and the reading and writing efficiency is improved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

fig. 1 is a schematic diagram of main steps of a method for processing data consistency in a distributed cluster according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a distributed database according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating a distributed transaction processing procedure according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a distributed transaction processing procedure according to another embodiment of the invention;

fig. 5 is a flowchart illustrating a method for processing data consistency in a distributed cluster according to another embodiment of the present invention;

fig. 6 is a schematic flowchart of a method for processing data consistency in a distributed cluster according to yet another embodiment of the present invention;

fig. 7 is a flowchart illustrating a method for processing data consistency in a distributed cluster according to yet another embodiment of the present invention;

FIG. 8 is a flowchart illustrating a data file merging method according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a processing apparatus for data consistency in a distributed cluster according to an embodiment of the present invention;

FIG. 10 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 11 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the embodiments of the present invention and the technical features of the embodiments may be combined with each other without conflict.

As shown in fig. 1, an embodiment of the present invention provides a method for processing data consistency in a distributed cluster, where the method is applied to any node in the distributed cluster, and may include the following steps S101 to S104:

step S101: acquiring a data processing request; the data processing request comprises a to-be-processed data identifier;

step S102: determining a data table to be processed, one or more data files corresponding to the data table to be processed and the transaction information of each data file according to the data identifier to be processed; the one or more data files are generated according to one or more transaction operations;

step S103: determining a query version number corresponding to the data table to be processed;

step S104: determining a target data file corresponding to the data processing request according to the transaction information and the query version number;

step S105: and processing the data corresponding to the to-be-processed data identification in the target data file.

The application scenario of the invention can be a scenario of data processing in a cloud-native distributed database, and the data processing process of the distributed database generally adopts a plurality of distributed nodes to respond to data processing requests so as to improve the concurrency performance of the distributed database. For any node in the distributed cluster, the data consistency processing method in the distributed cluster provided by the embodiment of the invention can be adopted to process the data.

In an embodiment of the present invention, processing the data corresponding to the to-be-processed data identifier in the target data file includes: and inquiring, updating, deleting or adding new data corresponding to the to-be-processed data identification, or merging the target data file. Each data processing request can be completed by one submitted transaction, namely, a plurality of different transactions respectively execute data processing requests for inquiring, updating, deleting or adding data.

Common architectures for distributed databases can be divided into SN (Share Nothing) framework and SD (Share Disk) framework. The SN frame means that all nodes of the distributed database do not share any resource, and each node performs data interaction through a network; while the SD framework refers to a distributed database in which all nodes share a single store. Generally, the distributed database native to the cloud adopts an SD framework. Fig. 2 is a schematic diagram of an SD shared storage architecture according to an embodiment of the present invention, where a client 1 and a client 2 are terminal applications for issuing data processing requests, for example, data processing requests for adding, updating, deleting, and querying a database may be issued; the computing node 1 and the computing node 2 … are typically virtual machines or container nodes on the cloud, the computing nodes are also nodes in the distributed cluster, and are used for processing the data processing requests, and the nodes all read data from the same shared storage or perform operations such as insertion, modification and deletion on the data in the same shared storage; shared storage typically provides an object storage service or a cloud file service on the cloud.

The bottom layer of the shared storage can adopt various storage modes, and in one embodiment of the invention, a merged tree storage structure can be adopted for data storage so as to obtain a new data file after each transaction operation.

The merged Tree, namely, the Merge Tree, is a storage structure similar to an LSM Tree (Log Structured Merge Tree). By adopting the Merge Tre mode, a new data file can be generated for newly added data of the data table every time, small files can be combined periodically according to the combination list to form a large file, and the method is suitable for scenes in which data are inserted in batches. The operation of updating or deleting data on the data table does not modify the original data file, the original data file is copied to generate a new data file, and the operation of updating or deleting is executed in the new data file, so that the data files of multiple versions can be generated, and a background can be set to regularly clean the data files corresponding to the old versions which are not used any more. The method can simultaneously carry out the operations of reading data and writing data without adding a distributed lock, avoids the operation conflict between reading and writing, improves the concurrency of reading and writing, and further improves the performance of distributed data.

It can be understood that each data file is generated by one or more transaction operations, and thus, for the same data in the same data table, there may be multiple different versions of data files corresponding to the same data, that is, when querying the same identifier of data to be processed, multiple versions of data files may be queried, thereby causing a problem of query data inconsistency. In order to solve the problem, in an embodiment of the present invention, the target data file may be determined according to the query version number corresponding to the data processing request and the transaction information of the data file, and query is performed from the target data file, so as to ensure consistency of the query data.

In one embodiment of the invention, the target data file may be determined in the following manner: determining latest transaction information from the transaction information corresponding to one or more data files; taking the latest transaction information as the current transaction information, and circularly executing the following steps until the target data file is determined: determining whether the query version number matches the current transaction information; if so, taking the data file corresponding to the current transaction information as the target data file; and if not, taking the previous transaction information of the current transaction information as the current transaction information.

Specifically, the pending data table may be determined according to the pending data identifier in the data processing request. For example, in fig. 2, the client needs to query all data corresponding to the data identifier a, and first needs to determine the to-be-processed data table a corresponding to the data identifier a. According to the storage structure of the Merge Tree, one or more data files can be determined according to the data table to be processed. It will be appreciated that data operations of a distributed database are typically performed by distributed transactions, i.e., transactional operations. A distributed transaction may correspond to an SQL statement, a group of SQL statements, or a program, the distributed transaction is typically managed by a transaction manager system, the distributed transactions are executed and committed in chronological order, and the transaction number of the committed transaction is recorded in the transaction manager. In one embodiment of the present invention, transaction information may be set for each data file, the transaction information being set according to a corresponding transaction operation.

When data corresponding to a certain data identifier is updated or deleted, an existing data file can be copied to generate a new data file, data is updated or deleted in the new data file, after the update or deletion is completed, a transaction number of the transaction operation is set in transaction information of the new data file to serve as a first transaction number, meanwhile, a second transaction number is set in the transaction information of the copied existing data file, and the second transaction number corresponds to the transaction operation for modifying the data file and is used for indicating the transaction operation for updating or deleting the data. Wherein, the latest transaction information in the transaction information of a plurality of data files can be determined by the latest first transaction number of the data files.

In FIG. 3, transaction 1 is the transaction corresponding to the SQL statement of the newly added data, and file1 is generated when data (1,2,3) is newly added; the transaction 2 is a transaction corresponding to the SQL statement of the update data, and file2 is generated; transaction 3 is the transaction corresponding to the SQL statement that deleted the data, resulting in file3. The transaction information of each data file includes a first transaction number and a second transaction number of the data file.

As in fig. 3, the same data a may exist in a plurality of data files of different transaction information, such as file1, file2, and file3; the transaction information of files file1, file2, and file3 consists of a first transaction number and a second transaction number. The format may be as follows: first transaction number-second transaction number. As in (1-2) of FIG. 3, 1 represents the first transaction number of the File1, and 2 represents the second transaction number of the File 1. The first transaction number is a transaction number corresponding to the transaction operation that generated the data file, and it is understood that the transaction number of the transaction operation may be set to be incremented along with the transaction operation time, and then the first transaction number in the transaction information of the plurality of data files may also be incremented, for example, the first transaction numbers of file1, file2, and file3 are incremented by 1,2, and 3.

In one embodiment of the present invention, whether the query version number matches the current transaction information is determined in the following manner: and under the condition that the first transaction number is the same as the query version number, determining that the query version number is matched with the current transaction information.

It is understood that the first transaction numbers of multiple data files corresponding to the same data table may be continuously incremented, for example, the continuity may be ensured by some transaction management modules, for example, the first transaction numbers of file1, file2 and file3 in fig. 3 are continuously incremented by 1,2 and 3.

FIG. 3 depicts the processing of the same data by three distributed transaction operations, as follows:

transaction 1: the specific data of the new data a, a is (1,2,3) and the transaction operation of plug-in (1,2,3) is executed in the data table a, according to the storage structure of the Merge Tree, a data file1 is newly added, the content of the file1 is (1,2,3), the transaction information of the file1 is set to (1-0), that is, the first transaction number is set to the transaction number "1", and the second transaction number can be set to 0 or null since the data file1 is newly created and has not been updated or deleted, so that the update or deletion operation can be ensured to always occur after the new data file is created.

Transaction 2: after transaction 1 is successfully committed, the transaction information of file1 is visible to transaction 2, transaction 2 is to execute a transaction operation of updating data a (1,2,3) to (4,5,6), namely updating a from (1,2,3) - > (4,5,6), at this time, file1 is not modified, file1 is copied to generate a new data file2, and updating is performed in file2, so that the content of file2 is (4,5,6), file1 is also the old content (1,2,3), and then the transaction information of file2 is set to (2-0), namely the first transaction number is set to "2" and the second transaction number can be set to 0 (representing infinity); after transaction 2 commits, the second transaction number of the old data file1 may also be modified from 0 to 2, such as transaction information (1-2), to indicate that the data of file1 has an updated operation in transaction number 2.

Transaction 3: the method is an operation of deleting 6 in the data a (4,5,6), namely deleting (4,5,6) - > (4,5), at this time, the updating operation of the transaction 2 is completed, the transaction 3 copies the file2 (2-0) to generate a new data file3 according to the latest first transaction number 2, the deletion is performed in the file3, the content of the file3 is (4,5), and the transaction information (3-0) is set for the file3, the first transaction number is 3, and the second transaction number is 0; after transaction 3 is successfully committed, (2-0) of file2 is modified to (2-3) to indicate that the data of file2 has an update operation in transaction 3. It will be appreciated that the delete operation is also actually an update operation on the data, and thus the second transaction number represents the transaction number to which the update or delete operation corresponds.

Taking the data processing request as an example of the query request, the principle of performing data query by using the query version number corresponding to the data processing request may be as shown in fig. 3. At the stage of execution of different transactions in fig. 3, determining the target data file when querying data can be divided into the following two cases:

one is that: when the query request occurs after the transaction 2 is submitted and the transaction 3 is not executed or the execution is not completed, the file3 is not generated at this time, and if the query version number corresponding to the query request (data processing request) is determined to be 2; the latest transaction information in the data files file1 and file2 is (2-0), the (2-0) is used as the current transaction information to be matched with the query version number, the first transaction number (2) in the current transaction information is determined to be the same as the query version number (2), then the data file2 with the first transaction number of 2 is the target data file, and data can be obtained by querying the file2 (2-0) (4,5,6).

The other is as follows: after the query request is submitted in the transaction 3, the latest transaction information in the data files file1, file2 and file3 is (3-0), the query version number corresponding to the query request (data processing request) is determined to be 2, the latest transaction information is used as the current transaction information to be matched with the query version number, the first transaction number (3) is different from the query version number (2) can be obtained, then the previous transaction information (2-3) can be used as the current transaction information to be matched with the query version number, the first transaction number (2) in the transaction information (2-3) is determined to be the same as the query version number (2), namely the first transaction number (2) of the file2 is the same as the query version number (2), so that the target data file can be determined to be file2, and data (4,5,6) can be obtained by querying in the file2 (2-3).

From both cases, when a query request occurs, whether the transaction 3 is executed or not and whether the transaction is finished or not, consistent data can be queried according to the query version number 2 (4,5,6). Therefore, when a plurality of distributed nodes inquire the same data processing request, consistent data can be inquired even if the inquiry time of each node is possibly different, and the consistency of the data inquired in a distributed manner is ensured.

In another embodiment of the present invention, the first transaction numbers of the multiple data files corresponding to the same data table may also be discontinuous, for example, in fig. 4, the first transaction numbers in the transaction information of the files file1, file2, and file5 are respectively 1,2, and 5, at this time, the query version number is 4, and the query version number is different from the first transaction numbers of the three data files, and may be determined by the following method provided by the embodiment of the present invention: under the condition that the first transaction number is not larger than the query version number, judging whether the second transaction number is larger than the query version number; if yes, determining that the query version number is matched with the current transaction information; if not, determining that the query version number is not matched with the current transaction information. That is, a judgment condition of "first transaction number < = query version number < second transaction number" may be determined, and whether the query version number matches the current transaction information may be determined according to the judgment condition.

It can be understood that if the query version number is smaller than the first transaction number of a certain data file, which means that the query operation does not read the data file generated later until the data file is generated; similarly, if the query version number is greater than the second transaction number of a certain data file, it means that after the query operation performs a transaction operation that updates or deletes the data corresponding to the certain data file, the data in the certain data file is expired during the query, and the certain data file should not be read.

Fig. 4 is a schematic diagram of a processing procedure of a distributed transaction according to another embodiment of the present invention, where transaction 1 is a transaction operation for executing new data B in data table B, that is, inserting (4,5,6), generating file1, and the transaction information is (1-0); the transaction 2 is a transaction operation for updating the data b (4,5,6) to (7,8,9), after the transaction 2 is submitted, the file2 is generated, the transaction information is (2-0), and the transaction information of the file1 is modified to (1-2); transaction 5 is a transaction operation for deleting 9 of data b (7,8,9), and the generated transaction information of file5 is (5-0); after transaction 5 commits successfully, the transaction information of file2 is modified from (2-0) to (2-5) to indicate that the data of file2 has an update or delete operation in transaction 5.

The first transaction numbers of the multiple data files are not continuous, that is, multiple transaction operations of multiple different data are sequentially increased on the base of the same transaction manager, for example, a transaction operation of data a and a transaction operation of data b are sequentially obtained from the same transaction manager according to an execution sequence, so that it may happen that multiple transaction operations of data a and multiple transaction operations of data b are executed in an interleaved manner, thereby resulting in a discontinuous transaction number. In fig. 4, the query version number is 4 and these first transaction numbers are all different, then the process of determining the target data file may be as follows:

if the query version is determined to be 4 before the transaction 5 is submitted by the data processing request (query request), then when the query of the distributed nodes is not finished yet, the transaction 5 is submitted and finished, and the file5 is generated, in order to ensure that the data obtained by the query of the distributed nodes is consistent, the condition that one part of nodes queries file1 or file2 and the other part of nodes queries file5 does not occur, and the query version number can be matched with the transaction information of the data files. Taking the latest transaction information (5-0) in the data files file1, file2 and file5 as the current transaction information, taking the first transaction number (5) as the query version number (4) differently, and determining that the query version number is not matched with the current transaction information if the first transaction number (5) is greater than the query version number (4), then taking the previous transaction information as the current transaction information, namely taking the transaction information (2-5) of file2 as the current transaction information, and then determining that the transaction information (2-5) of file2 is matched with the query version number (4) according to the judgment condition that the first transaction number < = query version number < the second transaction number, namely determining that the transaction information (2-5) of file2 is matched with the query version number (4), then determining that the target data file is file2, and querying from file2 (2-5) to obtain data (7,8,9).

If a data processing request (query request) occurs just after transaction 2 commits, then the delete transaction operation for transaction 5 has not yet occurred. The latest transaction information (2-0) in the data files file1 and file2 may be used as the current transaction information, the first transaction number < query version number, that is, the query occurs after the transaction 2, and the second transaction number is 0 (infinity), that is, the query version number < the second transaction number may be considered, which means that no transaction operation for updating or deleting the data b has occurred yet, then the current transaction information (2-0) is matched with the query version number (4), that is, the data b is the latest value that can be read by the query request in the file2, the target data file may be determined to be file2, and the data (7,8,9) is obtained by querying from the file2 (2-0).

It is understood that in fig. 4, consistent data (7,8,9) can be queried according to the query version number 4. Therefore, when a plurality of distributed nodes inquire the same data processing request, consistent data can be inquired even if the inquiry time of each node is possibly different, and the consistency of the data inquired in a distributed manner is ensured.

In an embodiment of the present invention, for a case that the first transaction number of the multiple data files is continuously incremented, when the current latest first transaction number is not greater than the query version number, the current latest first transaction number may be sequentially compared with the query version number and the first transaction number in the multiple transaction information, and a data file corresponding to the first transaction number that is the same as the query version number is determined, so as to determine the data file as the target data file, as shown in fig. 3, which is a query process with a query version number of 2; when the current latest first transaction number is smaller than the query version number, it is assumed in fig. 3 that the query request is found after the transaction 3 has been submitted and the query version number is 4, then the data file3 corresponding to the latest first transaction number (3) may be used as the target data file. It can be understood that, in the case that the first transaction number of the plurality of data files is continuously incremented, whether the query version number matches the transaction information may be determined by the first transaction number, and then the second transaction number of the data files may not be set, or may be both set to 0 or null.

In an embodiment of the present invention, the determining the query version number corresponding to the to-be-processed data table includes: determining whether a transaction number corresponding to the data table to be processed exists in a cache; if yes, the transaction number is used as the query version number; if not, extracting the latest transaction number from the transaction manager, and taking the latest transaction number as the query version number.

For example, the same data table may be queried for the same query statement many times, in order to ensure that the same data file used for querying a plurality of data tables in the same query statement is the same, a table cache at the query statement level may be set, and when the statement is finished, the table cache is cleared. The table cache stores the identifier of the data table to be processed and the determined transaction number (also referred to as the determined query version number) of the data table when the data table is queried for the first time in the current query statement, and at this time, if the transaction number corresponding to the data table to be processed is found in the table cache, the transaction number can be directly used as the query version number in subsequent queries, so that the time for determining the query version number can be greatly saved, and the query efficiency is improved. And if the cache is emptied and the transaction number is not found, acquiring the latest submitted transaction number from the transaction manager, and taking the latest transaction number as the query version number.

After the query version number is determined, the query is carried out, and the data file of the fixed version can be ensured to be queried each time, so that the condition that old data and newly updated data are queried at the same time can be avoided, the consistency of the query data of a plurality of distributed nodes can be ensured, and the data consistency of a distributed database can be ensured; in addition, the query can occur at any time, and whether data is newly added, updated or deleted at present does not need to be considered, so that a distributed lock does not need to be additionally added between reading and writing, the reading and writing concurrency capability of the distributed database is greatly improved, and the performance of the distributed database is improved.

When a plurality of nodes in the distributed cluster perform distributed query, before processing the data corresponding to the to-be-processed data identifier in the target data file, any one node may adopt the following manner provided by an embodiment of the present invention: determining that each node in the distributed cluster has synchronized the target data file.

In an embodiment of the present invention, processing the data corresponding to the to-be-processed data identifier in the target data file further includes: and merging the target data files. Due to the adoption of the Merge Tree storage structure, a large number of small data files can be generated when a new data file is generated every time data is inserted, and the small data files can be combined periodically to release the disk space.

In an embodiment of the present invention, processing the data corresponding to the to-be-processed data identifier in the target data file includes: and inquiring, updating, deleting or newly adding the data corresponding to the to-be-processed data identifier.

A data method provided by the embodiment of the present invention is described in detail below with reference to fig. 5, fig. 6, and fig. 7, which take query data, new data, and updated data as examples.

Fig. 5 is a schematic flowchart of a processing method for data consistency in a distributed cluster according to another embodiment of the present invention, which is mainly a process of querying data in a distributed database, and includes the following steps:

step S501: after acquiring a data query request, starting query, and performing query optimization, wherein the query optimization comprises steps S502 to S507; query optimization is the process of determining the query version number, i.e., the query version number.

Step S502: determining a look-up table; namely, the data table to be processed corresponding to the data identifier to be processed is determined.

Step S503: the query version number is obtained, and execution is started from S504.

Step S504: judging that the cache has a table version number; if yes, directly reading the table version number and taking the table version number as a query version number, and continuing to execute S507; if not, step S505 is performed.

Step S505: a transaction number is used; the latest transaction number submitted is obtained from the transaction manager and used as the query version number.

Step S506: adding one to the number of references;

a reference count may be set for each transaction in the transaction manager; and adding 1 to the reference times corresponding to the transaction number used in the query, and executing the step S507.

If the cache has the table version number, the table version number indicates that the same table is queried in one query statement, the reference times are increased, and the reference times are not increased.

Step S507: and determining a target data file.

Step S508: executing the query; including steps S509 to S512.

Step S509: the process of reading the stored data includes steps S510 to S512.

Step S510: rewriting the query statement and adding a query version number; that is, the query version number is put into a query statement so as to specify a data file corresponding to a specific query version number during query.

Step S511: a distribution node; that is, the target data file obtained in the query optimization process is distributed to different distributed nodes for query, so that multiple distributed nodes execute consistency query.

The distributed cache stores the version number of the data file (i.e. the first transaction number in the transaction information) that has completed synchronization, and the data files before the version number are all synchronized. For example, the version number stored in the distributed cache is 8, when a version 9 is synchronized, the version number in the cache is updated to 9, if the synchronized version is 12, the version number is recorded, the version number in the cache is not updated, and the version number in the cache is not updated to 12 until all the versions 9, 10, and 11 are updated.

The distributed nodes read the version numbers stored in the caches from the distributed caches, judge whether the query version number determined at present is smaller than or equal to the stored version number, if so, indicate that the metadata of the target data file is synchronously completed in the distributed caches, and can directly read the content of the target data file from the distributed caches without reading from a shared storage disk, so that the query speed is accelerated, and the query efficiency is improved. If the query version number determined currently is greater than the minimum version number, it indicates that the data synchronization is not completed, and may wait for the data of the target data file to be synchronized in the distributed cache and then read the data from the distributed cache, or may directly read the data from the shared storage without waiting.

When a plurality of distributed nodes are used for consistency query, consistency hash can be carried out according to the number of a target data file, the data file with the hash value equal to the number of the current node is queried by the current node. Thus, the whole query is divided into N parts, and each node queries a part of data. Consistent hashing ensures that the target data file set seen by all nodes is consistent, otherwise, it cannot be ensured that the data file subsets queried by all nodes are different, and the aggregated data file set is exactly the whole data file set, which is ensured by the minimum version number stored in the distributed cache of consistent querying. For example, the number of the file in the target data file set involved in the query is {1,2,3,4,5,6,7}, and there are currently 3 nodes (numbering from 0). We use hash function F (X) = X%3; then the {1,4,7} file is obtained for access by node 1, {2,5} file for access by node 2, and {3,6} file for access by node 0. And the distributed nodes respectively carry out inquiry according to the determined file subsets.

Before a plurality of distributed nodes perform the consistency query, steps S501 to S511 are generally performed by one initial node to complete the preparation work before distribution.

Step S512: merging all results; and after the plurality of distributed nodes are inquired, combining a plurality of results, returning the results to the initial node, and sending the results to the client side which provides the data processing request.

Step S513: the number of references is reduced by one; and after the query is finished, reducing the reference times of the transaction number corresponding to the query version number by one.

Since multiple data files are generated during the write operation (update, delete, and insert) of data, some old versions of data files are no longer queried, e.g., file1 in fig. 3 is no longer used after transaction 3, and these old data files can be cleaned up periodically to release the disk space; cleaning can be carried out according to the reference times of the submitted transaction number, and the process is as follows: obtaining the maximum transaction number Y with the reference times of 0, namely sequencing all the transaction numbers with the reference times of 0 to obtain the maximum value Y; and traversing all the data files, and deleting the files of which the second transaction number is not more than Y. This process can be put in the background processing without affecting the execution of the distributed data processing.

Fig. 6 is a flowchart illustrating a method for processing data consistency in a distributed cluster according to another embodiment of the present invention; the process of adding (inserting) data in the distributed database mainly includes the following steps:

step S601: acquiring a data insertion request, starting insertion, and performing lexical analysis on a sentence; performing lexical analysis on the SQL statement; the flow of data insertion is executed by one insertion node, and after the insertion is completed, data synchronization is performed with other nodes.

Step S602: syntax analysis of the sentence is carried out; and carrying out syntax analysis on the SQL statement.

Step S603: optimizing the query; namely, SQL statements are optimized, a data table to be inserted is determined, and at the moment, the corresponding data insertion request contains data to be inserted.

Step S604: performing the insertion may include steps S605 to S610.

Step S605: the writing of the data file is started.

Step S606: writing a temporary file; and writing the data to be inserted into the temporary file.

Step S607: renaming the file; the temporary file is renamed to a new data file.

Step S608: setting transaction information; acquiring a transaction number of a current insertion transaction, and taking the transaction number of the insertion transaction as a first transaction number; because the newly created data file is used, the second transaction number may be set to 0, may not be set, or may be set to other specific characters, so as to facilitate comparison as infinity when subsequently comparing with the query version number.

Step S609: submitting the transaction; that is, it is determined that the insertion of the data is completed, after the completion of the submission, the insertion node writes the data into a disk of the shared storage, and sends the latest data file name (e.g., file 1) to other nodes, and the other nodes read the disk file from the shared storage to acquire and update the latest data file information, thereby implementing the synchronization information. In order to guarantee the performance of inserting data, other nodes are not waited to read the new data file in the transaction submitting phase, and the new data information is asynchronously synchronized to other nodes in the background.

Step S610: modifying related data file information in the memory information; it may be to modify the data file information stored in the distributed memory or distributed cache.

Fig. 7 is a flowchart illustrating a method for processing data consistency in a distributed cluster according to yet another embodiment of the present invention; the process of updating data in a distributed database mainly comprises the following steps:

step S701: acquiring the update start of a data update request, and performing lexical analysis on a sentence; performing lexical analysis on the SQL sentence; in order to avoid the situation that data are inconsistent due to simultaneous data update by multiple nodes, a distributed lock needs to be added when data are updated, and update operations are sequenced, for example, transaction 2 and transaction 3 in fig. 3 are sequentially executed, and a delete operation of transaction 3 is also equivalent to another form of data update; and updating data by the updating node which acquires the distributed lock.

Step S702: syntax analysis of the sentence is carried out; and carrying out syntax analysis on the SQL statement.

Step S703: query optimization, in which a target data file corresponding to a data identifier to be updated needs to be determined before updating data, and the process is also similar to the query optimization in fig. 4, and is not described herein again. For example, transaction 3 in FIG. 3 needs to determine whether to delete file1 or file2 when performing data deletion.

Step S704: the performing of the update includes steps S705 to S712.

Step S705: the data file is updated.

Step S706: copying the temporary file; and determining a target data file, and copying the target data file into a temporary file.

Step S707: updating the temporary file; and updating data in the temporary file.

Step S708: renamed to a new data file; the temporary file is renamed to a new data file.

Step S709: setting transaction information of a new data file; acquiring the transaction number of the current update transaction, and taking the transaction number of the update transaction as the first transaction number of the new data file; because the newly created data file is used, the second transaction number may be set to 0, may not be set, or may be set to other specific characters, so as to facilitate comparison as infinity when subsequently comparing with the query version number, as in fig. 3, the file2 transaction information is set to (2-0).

Step S710: setting transaction information of an old data file; and modifying the second transaction number of the original target data file (old data file) into the transaction number of the update transaction, as shown in fig. 3, modifying the transaction information of the setting file1 from (1-0) to (1-2).

Step S711: submitting the transaction; namely, the updated data is determined to be completed, after the updated data is submitted, the updated node writes the data into the disk and synchronizes to other nodes, and the other nodes synchronize information and read disk files.

Step S712: and modifying the related data file information in the memory information.

The processes of steps S711 and S712 are the same as the process of synchronizing information and the process of modifying memory information in fig. 6, and are not further described herein

In addition, the data deleting operation is to copy the target data file into a temporary file, delete data in the temporary file, rename the temporary file into a new data file, and set transaction information between the new data file and the original target data file, which is similar to the data updating operation in fig. 6 and is not repeated here.

In one embodiment of the invention, processing the target data file further comprises merging the target data file. Fig. 8 is a flowchart illustrating a data file merging method according to an embodiment of the present invention, which may include the following steps:

step S801: starting merging, and acquiring a list to be merged; the storage mechanism of the Merge Tree will regularly sort multiple small data files and then get a list of files to be merged. It can be understood that the merge file operation also needs to add a distributed lock to ensure that there is no conflict with the update transaction operation, and the merge node that acquires the distributed lock executes the merge process.

Step S802: executing merging; including steps S803 to S810.

Step S803: reading a file to be merged; and circularly reading the files in the file list to be merged.

Step S804: data is merged into a temporary file; and newly building a temporary file, and writing the data in the file to be merged into the temporary file.

Step S805: judging that the list to be merged is empty; if yes, step 806 is executed, if no, the operation returns to step 803, and the next file to be merged is read continuously.

Step S806: renamed to a new data file; and after the data of the data files in the list to be merged are all written into the temporary file, renaming the temporary file to be a new data file.

Step S807: setting transaction information of a new data file; and acquiring the transaction number of the current merging transaction, taking the transaction number as the first transaction number of the new data file, and setting the second transaction number to be 0.

Step S808: setting transaction information of files to be merged; and setting the second transaction numbers of the files to be merged in the list to be merged as the transaction number of the merged transaction.

Step S809: submitting the transaction; namely, the merged file is determined to be completed, after the merged file is submitted, the merged node sends the new data file name to other nodes for synchronization, and the other nodes synchronize information and read the disk file.

Step S810: and modifying the related data file information in the memory information.

The processes of steps S809 and S810 are the same as the process of synchronizing information and the process of modifying memory information in fig. 6, and are not described herein again.

According to the method for processing the data consistency in the distributed cluster, provided by the embodiment of the invention, when any node in the distributed cluster is processed according to the same data processing request, the same target data file can be determined according to the query version number and the transaction information of the data file; therefore, when the distributed query is carried out, all the nodes can query according to the same target data file, so that the condition that the nodes read the same numerical value of the same data is ensured, and the data consistency of the distributed query is further ensured.

As shown in fig. 9, an embodiment of the present invention provides a processing apparatus 900 for data consistency in a distributed cluster, including: an acquisition module 901, a determination module 902 and a processing module 903; wherein the content of the first and second substances,

the obtaining module 901 is configured to obtain a data processing request; the data processing request comprises a to-be-processed data identifier;

the determining module 902 is configured to determine, according to the to-be-processed data identifier, a to-be-processed data table, one or more data files corresponding to the to-be-processed data table, and transaction information of each of the data files; the one or more data files are generated according to one or more transaction operations; determining a query version number corresponding to the data table to be processed; determining a target data file corresponding to the data processing request according to the transaction information and the query version number;

the processing module 903 is configured to process data corresponding to the identifier of the to-be-processed data in the target data file.

In an embodiment of the present invention, the determining module 902 is configured to determine latest transaction information from the transaction information corresponding to one or more data files; taking the latest transaction information as the current transaction information, and circularly executing the following steps until the target data file is determined: determining whether the query version number matches the current transaction information; if so, taking the data file corresponding to the current transaction information as the target data file; and if not, taking the previous transaction information of the current transaction information as the current transaction information.

In an embodiment of the present invention, the determining module 902, configured to determine whether the current transaction information includes a first transaction number, where the first transaction number corresponds to a transaction operation for generating the data file, and the determining whether the query version number matches the current transaction information includes: and under the condition that the first transaction number is the same as the query version number, determining that the query version number is matched with the current transaction information.

In an embodiment of the present invention, the determining module 902 is configured to further include a second transaction number in the current transaction information, where the second transaction number corresponds to a transaction operation for modifying the data file; the determining whether the query version number matches the current transaction information includes: under the condition that the first transaction number is not larger than the query version number, judging whether the second transaction number is larger than the query version number; if yes, determining that the query version number is matched with the current transaction information; if not, determining that the query version number is not matched with the current transaction information.

In an embodiment of the present invention, the determining module 902 is configured to determine whether a transaction number corresponding to the to-be-processed data table exists in a cache; if yes, the transaction number is used as the query version number; if not, extracting the latest transaction number from the transaction manager, and taking the latest transaction number as the query version number.

In an embodiment of the present invention, the determining module 902 is configured to determine that the to-be-processed data table uses a merge tree storage structure for data storage, so as to obtain a new data file after each transaction operation.

In an embodiment of the present invention, the determining module 902 is configured to determine that each node in the distributed cluster synchronizes the target data file before the processing of the data in the target data file corresponding to the identifier of the data to be processed.

In an embodiment of the present invention, the processing module 903 is configured to perform query, update, deletion, or addition processing on data corresponding to the to-be-processed data identifier, or perform merging processing on the target data file.

According to the processing device for data consistency in the distributed cluster, provided by the embodiment of the invention, when any node in the distributed cluster is processed according to the same data processing request, the same target data file can be determined according to the query version number and the transaction information of the data file; therefore, when the distributed query is carried out, all the nodes can query according to the same target data file, so that the condition that the nodes read the same numerical value of the same data is ensured, and the data consistency of the distributed query is further ensured.

Fig. 10 shows an exemplary system architecture 1000 to which a method for processing data consistency in a distributed cluster or a device for processing data consistency in a distributed cluster according to an embodiment of the present invention may be applied.

As shown in fig. 10, the system architecture 1000 may include

terminal devices

1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the

terminal devices

1001, 1002, 1003 and the server 1005. Network 1004 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

1001, 1002, 1003 to interact with a server 1005 via a network 1004 to receive or transmit messages or the like.

The

terminal devices

1001, 1002, 1003 may be various electronic devices having a display screen and supporting information browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 1005 may be a server that provides data processing, such as a background management server that provides support for data processing requests made by users using the

terminal devices

1001, 1002, and 1003. The background management server can analyze and process the received data such as the data processing request and feed back the processing result to the terminal equipment.

It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 1005, and accordingly, the data processing apparatus is generally disposed in the server 1005.

It should be understood that the number of terminal devices, networks, and servers in fig. 10 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 11, shown is a block diagram of a computer system 1100 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 11 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 11, the computer system 1100 includes a Central Processing Unit (CPU) 1101, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 1102 or a program loaded from a storage section 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the system 1100 are also stored. The CPU 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

The following components are connected to the I/O interface 1105: an input portion 1106 including a keyboard, mouse, and the like; an output portion 1107 including a signal output unit such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 1108 including a hard disk and the like; and a communication section 1109 including a network interface card such as a LAN card, a modem, or the like. The communication section 1109 performs communication processing via a network such as the internet. A driver 1110 is also connected to the I/O interface 1105 as necessary. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1110 as necessary, so that a computer program read out therefrom is mounted into the storage section 1108 as necessary.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 1109 and/or installed from the removable medium 1111. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 1111.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises an obtaining module, a determining module and a processing module. The names of these modules do not in some cases constitute a limitation to the module itself, and for example, the acquiring module may also be described as a "module that acquires a data processing request".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring a data processing request; the data processing request comprises a to-be-processed data identifier; determining a data table to be processed, one or more data files corresponding to the data table to be processed and the transaction information of each data file according to the data identifier to be processed; the one or more data files are generated according to one or more transaction operations; determining a query version number corresponding to the data table to be processed; determining a target data file corresponding to the data processing request according to the transaction information and the query version number; and processing the data corresponding to the to-be-processed data identifier in the target data file.

According to the technical scheme of the embodiment of the invention, when any node in the distributed cluster is processed according to the same data processing request, the same target data file can be determined according to the query version number and the transaction information of the data file; therefore, when distributed query is carried out, all the nodes can carry out query according to the same target data file, so that the condition that the nodes read the same numerical value of the same data is ensured, and the data consistency of the distributed query is further ensured.

Furthermore, the data table is stored by using the merged tree storage structure, a new data file is generated every time of data writing operations such as adding, updating or deleting data, and the data in the new data file is written, so that data files of multiple versions are obtained, the write operation can be prevented from influencing the read operation, the read-write concurrency of the distributed database is improved, and the performance of the distributed database is further improved; in addition, a plurality of data files are combined regularly, the situation that a large number of small files appear is avoided, storage space is saved, and therefore reading and writing efficiency is improved.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing data consistency in a distributed cluster is applied to any node in the distributed cluster, and includes:

2. The method according to claim 1, wherein the determining a target data file corresponding to the data processing request according to the transaction information and the query version number comprises:

3. The method of claim 2, wherein the current transaction information includes a first transaction number, wherein the first transaction number corresponds to a transaction operation that generated the data file, and wherein determining whether the query version number matches the current transaction information comprises:

4. The method of claim 3, wherein the current transaction information further includes a second transaction number, the second transaction number corresponding to a transaction operation that modifies the data file; the determining whether the query version number matches the current transaction information includes:

5. The method according to claim 1, wherein the determining a query version number corresponding to the to-be-processed data table includes:

determining whether a transaction number corresponding to the data table to be processed exists in a cache;

if yes, the transaction number is used as the query version number;

if not, extracting the latest transaction number from the transaction manager, and taking the latest transaction number as the query version number.

6. The method of claim 1, wherein the data table to be processed is stored in a merged tree storage structure to obtain a new data file after each transaction operation.

7. The method according to claim 1, wherein before the processing the data corresponding to the to-be-processed data identifier in the target data file, the method comprises:

8. The method according to claim 1, wherein the processing the data in the target data file corresponding to the to-be-processed data identifier includes:

9. A device for processing data consistency in a distributed cluster, comprising: the device comprises an acquisition module, a determination module and a processing module; wherein the content of the first and second substances,

the acquisition module is used for acquiring a data processing request; the data processing request comprises a to-be-processed data identifier;

10. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.