CN114942955A - Data export method, device, export node, medium and system - Google Patents

Abstract

The embodiment of the invention discloses a data export method, a data export device, an export node, a medium and a system. The method comprises the following steps: when an export instruction is received, acquiring a target global transaction version number; determining at least one target fragment according to the export instruction; and creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task. According to the method, the query task containing the target global transaction version number is created for the determined target fragment, data export operation is performed on the basis, data corresponding to the target global transaction version number can be exported, the problem that the exported data in the online scene of the distributed database are inconsistent is solved, and the reliability of data export is improved.

Description

Data export method, device, export node, medium and system

Technical Field

The embodiment of the invention relates to the technical field of data processing, in particular to a data export method, a data export device, an export node, a medium and a system.

Background

The distributed database is composed of a plurality of physically independent database servers, and each database server is a database node.

In many data processing scenarios, a user is often required to first derive the required data from a distributed database storing a large amount of data. However, in the process of exporting required data online in a distributed database scenario, because data in the distributed database changes in real time, the data exported from the distributed database based on the existing exporting method may have a problem that the exported data is inconsistent, thereby affecting the reliability of data export.

Disclosure of Invention

The embodiment of the invention provides a data export method, a data export device, an export node, a medium and a system, which are used for solving the problem of inconsistent online exported data in a distributed database scene and improving the reliability of data export.

According to an aspect of the embodiments of the present invention, there is provided a data export method, including:

when an export instruction is received, acquiring a target global transaction version number;

determining at least one target fragment according to the export instruction;

and creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task.

According to another aspect of the embodiments of the present invention, there is provided a data exporting apparatus, including:

the obtaining module is used for obtaining a target global transaction version number when the export instruction is received;

a determining module, configured to determine at least one target segment according to the derivation instruction;

and the execution module is used for creating a corresponding query task for the target fragment based on the target global transaction version number and executing data export operation based on the query task.

According to another aspect of the embodiments of the present invention, there is provided a derivation node, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform a data derivation method according to any of the embodiments of the present invention.

According to another aspect of the embodiments of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data export method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, firstly, when a export instruction is received, a target global transaction version number is obtained; then determining at least one target fragment according to the export instruction; and finally, creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task. According to the scheme, the query task containing the target global transaction version number is created for the determined target fragment based on the obtained target global transaction version number, data export operation is performed on the basis, data corresponding to the target global transaction version number can be exported, the problem that the exported data of the distributed database is inconsistent in an online scene is solved, and the reliability of data export is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a flowchart of a data export method according to an embodiment of the present invention;

fig. 2 is a flowchart of a data export method according to a second embodiment of the present invention;

fig. 3 is a schematic diagram illustrating an implementation of a data export method according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of a data deriving device according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of a derived node according to a fourth embodiment of the present invention;

fig. 6 is a schematic structural diagram of a data export system according to a fifth embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," "target," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The export of a data table may be performed through multiple batch query operations, the data is real-time at each export, and the data in the data file is not at the same time after the export task is completed, because it is possible that the previously exported data is modified during the export process, i.e. is not finally consistent. When a batch export query operation is executed once, there is a possibility that data on each database fragment is not globally consistent, and export data of an export file is inconsistent.

The embodiment of the invention provides a distributed transaction strong consistency scheme based on a global transaction version number, which can adopt a unique and increasing global transaction version number as a transaction mark and adopts a database two-stage submission method to ensure that the global transactions of a distributed database are consistent.

Specifically, this embodiment proposes a two-phase commit protocol method based on a global transaction version number, which allocates a global transaction version number to a transaction when the transaction starts; when the transaction satisfies the commit state, enter the PREPARE phase (i.e., XA PREPARE); and entering a COMMIT stage (namely XA COMMIT) when the transaction is committed, and carrying a committed global transaction version number so as to ensure the consistency of reading and writing among the transactions. The distributed database after the method is adopted can also be understood as a distributed database with a global transaction version number.

On this basis, the embodiment of the present invention provides a data export method, so that in a distributed database scenario, some databases or some data tables are exported online, so that data in an export file is globally transaction-consistent. The method lays a foundation for the global consistency of the imported data files, namely the data of each fragment is globally consistent at a certain moment after being imported.

Example one

Fig. 1 is a flowchart of a data export method provided in an embodiment of the present invention, where this embodiment is applicable to a case of exporting data in an online scenario of a distributed database, and the method may be executed by a data export apparatus, where the data export apparatus may be implemented in the form of hardware and/or software, and the data export apparatus may be configured in an export node, where the export node includes, but is not limited to: a server, a computer, a notebook computer, or a tablet computer. As shown in fig. 1, the method includes:

and S110, acquiring a target global transaction version number when the export instruction is received.

In the present embodiment, the export node may be understood as a node for receiving an export instruction and processing data export and the like. An export instruction may be understood as an instruction for triggering the export of data. The target global transaction version number may be understood as a global transaction version number to be used by the export instruction.

When the export node receives the export instruction, the export node indicates that corresponding data is needed to be exported currently, and at the moment, the target global transaction version number can be obtained so as to be used for subsequent data export. How to obtain the target global transaction version number is not particularly limited, for example, the corresponding target global transaction version number may be obtained by requesting the global transaction version management node responsible for managing the global transaction version number.

Optionally, the obtaining the target global transaction version number includes: sending an acquisition request to a global transaction version management node, wherein the acquisition request is a request for acquiring a target global transaction version number; and receiving a target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node according to the current global transaction version number.

Wherein a global transaction version management node is understood to be a management node for maintaining and distributing global transaction version numbers. A get request may be understood as a request to get a target global transaction version number. Global Transaction version number (STS) may refer to a unique incrementing Sequence issued by a global Transaction version management node to characterize a timestamp Snapshot at a time in a distributed database. The global transaction version number can be composed of physical time and logical time, wherein the physical time is millisecond difference time from 00 hours to 00 minutes and 00 seconds on 01-1970, and the logical time is a sequence value for distinguishing different transactions at the same physical time; the specific composition of the global transaction version number is not particularly limited herein.

Specifically, when receiving a derivation instruction, a derivation node sends an acquisition request for acquiring a target global transaction version number to a global transaction version management node; the global transaction version management node receives an acquisition request sent by the export node, determines a target global transaction version number according to the current global transaction version number, and returns the target global transaction version number to the export node; and the export node receives the target global transaction version number returned by the global transaction version management node. In this embodiment, how to determine the target global transaction version number according to the current global transaction version number is not specifically limited, for example, when the global transaction version management node receives an acquisition request sent by the export node, the current global transaction version number is used as the target global transaction version number; where the current global transaction version number can be understood as a time-stamped snapshot of the current time.

And S120, determining at least one target fragment according to the export instruction.

In this embodiment, the distributed database may include a plurality of database shards, and data stored in the same database table may be stored in the plurality of database shards in a distributed manner through a certain specific condition, that is, each database shard may store a part of data in one database table. Here, how to dispersedly store the data in one database table on a plurality of database shards is not specifically limited, and for example, the data in one database table may be dispersedly stored on all database shards in the distributed database, or the data in one database table may be dispersedly stored on part of database shards in the distributed database.

A target partition may be understood as a database partition to which export instructions correspond to data to be exported. There is no specific limitation on how to determine the at least one target slice according to the derivation instruction; for example, if the data to be exported corresponding to the export instruction is distributed on each database fragment in the distributed database, the target fragment may be all database fragments in the distributed database; and if the data to be exported corresponding to the export instruction is distributed on part of the database fragments in the distributed database, the part of the database fragments are the target fragments. Wherein, the data to be exported can be understood as the data to be exported corresponding to the export instruction. It will be appreciated that the data to be exported on each target slice may be different.

S130, creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task.

In this embodiment, a query task may be understood as a task for querying data to be derived distributed on a target slice. The created query task may include a target global transaction version number, so that when a data export operation is subsequently performed according to the query task, data corresponding to the target global transaction version number is exported. A data export operation may be understood as an operation that exports data to be exported that corresponds to an export instruction.

There is no specific limitation on how the exporting node creates the corresponding query task for the target segment based on the target global transaction version number. If so, respectively creating a corresponding query task for each determined target fragment based on the target global transaction version number to query the data to be exported corresponding to the export instructions distributed on each target fragment; on this basis, for each target fragment, if the data size of the data to be exported distributed on the target fragment is large (e.g., exceeds a preset data size threshold), the query task corresponding to the target fragment may be divided into multiple batches of queries (i.e., divided into multiple sub-query tasks), where the multiple sub-query tasks may be executed in parallel, and each sub-query task may include a target global transaction version number. Correspondingly, if the data size of the data to be exported distributed on the target fragment is not large (e.g. within the preset data size threshold), the query task can be directly executed without batch query, and the divided sub-query task can be understood as the query task itself.

After creating a corresponding query task for each target segment, if one query task is divided into a plurality of sub-query tasks, a corresponding query command can be created for each sub-query task; if one query task is not queried in batches, the sub-query task can be regarded as the query task, and a corresponding query command is directly created for the query task. On this basis, the export node sends the created query command to the corresponding target segment, and the query command can be understood as a query statement command containing a target global transaction version number generated based on the sub-query task, that is, each query command contains the target global transaction version number. After receiving the query command of the export node, the target fragment executes query on the data to be exported stored in the target fragment according to the query command, and returns the query result to the export node. And after receiving the query result of the target fragment, the export node processes the query result and exports the processed data result. How the export node processes the query result is not particularly limited, and may be, for example, processing the query result according to a uniform data storage format so that the query result is converted into the uniform data storage format for export.

It should be noted that, in the case of multiple sub-query tasks, the sub-query tasks may be executed in parallel. All the sub-query tasks can be executed together in parallel, or part of the sub-query tasks are executed in parallel first, and when the part of the sub-query tasks are executed, part of the sub-query tasks in the rest sub-query tasks are executed in parallel, and so on until all the sub-query tasks are executed.

The data export method provided by the embodiment of the invention comprises the steps of firstly, acquiring a target global transaction version number when an export instruction is received; then determining at least one target fragment according to the export instruction; and finally, creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task. According to the method, the query task containing the target global transaction version number is created for the determined target fragment based on the acquired target global transaction version number, data export operation is performed on the basis, data corresponding to the target global transaction version number can be exported, the problem that the exported data of the distributed database is inconsistent in an online scene is solved, and the reliability of data export is improved.

Example two

Fig. 2 is a flowchart of a data export method according to a second embodiment of the present invention, which is further detailed based on the above embodiments. In this embodiment, a process of determining at least one target fragment according to a derivation instruction, creating a corresponding query task for the target fragment based on the target global transaction version number, and performing a data derivation operation based on the query task is specifically described. It should be noted that technical details that are not described in detail in the present embodiment may be referred to any of the above embodiments. As shown in fig. 2, the method includes:

s210, when the export instruction is received, the target global transaction version number is obtained.

In this embodiment, when receiving the export instruction, the export node may send an obtaining request for obtaining a target global transaction version number to the global transaction version management node, and receive a target global transaction version number returned by the global transaction version management node, where the target global transaction version number may be determined by the global transaction version management node according to the current global transaction version number.

And S220, analyzing the export instruction to obtain the fragment where the data to be exported is located.

In this embodiment, the export node parses the obtained export instruction. So as to obtain the slice where the data to be exported corresponding to the export instruction is located. How to parse the export instruction is not particularly limited herein.

And S230, taking the fragments where the data to be exported are located as target fragments, wherein the number of the target fragments is at least one.

In this embodiment, the export node may use the segment where the data to be exported is located as the target segment, where the number of the target segments may be at least one.

S240, creating an inquiry task containing the target global transaction version number for the target fragment.

In this embodiment, the export node may create a corresponding query task including a target global transaction version number for each target segment.

And S250, dividing the query task into at least one sub-query task according to the distribution information of the data to be exported on the target fragment.

In this embodiment, the distribution information may be understood as data amount information of the data to be exported distributed on the target slice. A sub-query task may be understood as dividing a query task into multiple batches of queries, each batch querying a corresponding task. According to the distribution information of the data to be exported on the target fragment, the query task can be divided into at least one sub-query task. The sub-query tasks can contain target global transaction version numbers, and the sub-query tasks can be executed in parallel.

S260, creating a corresponding query command for each sub-query task in the query task, and sending the query command to the corresponding target fragment.

In this embodiment, the query command may be understood as a query statement command containing a target global transaction version number generated based on the sub-query task. And the export node creates a corresponding query command for each sub-query task in the query tasks and sends the query command to the corresponding target fragment.

And S270, receiving a query result returned by the target fragment.

In this embodiment, the query result may be understood as target to-be-exported data corresponding to a target global transaction version number determined by the target segment according to the query command. The target data to be exported can be understood as data to be exported, which is obtained by the target fragment according to the corresponding query command. And the export node receives the query result returned by each target fragment.

Optionally, the query result includes target data to be exported, which is determined by the target segment according to the query command and corresponds to the target global transaction version number.

How to determine the target data to be exported corresponding to the target global transaction version number according to the query command is not specifically limited. For example, the target data to be exported may be a data result corresponding to a target global transaction version number obtained by the target shard through executing the query command.

And S280, processing the query result and exporting a processing result.

In this embodiment, the processing result may be understood as a data result obtained by processing the query result. The export node processes the query result, for example, the data storage format of the query result can be uniformly processed according to a set format, and the obtained result is the processing result; on this basis, after obtaining and processing the query results of all target segments, the export node may export the processing result, for example, export the processing result to a corresponding export file, which is not limited herein.

The data export method provided by the second embodiment of the invention embodies the processes of determining at least one target fragment according to the export instruction, creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task. The method includes the steps of creating an inquiry task containing a target global transaction version number for a determined target fragment based on the target global transaction version number, dividing the inquiry task into at least one sub-inquiry task, creating a corresponding inquiry command for each sub-inquiry task on the basis, executing data export operation based on the inquiry commands, exporting data corresponding to the target global transaction version number, solving the problem that online exported data under a distributed database scene are inconsistent, and improving reliability of data export. In addition, the data storage format of the processed query result can be unified by processing the query result returned by the target fragment.

The present invention is exemplified below.

Fig. 3 is a schematic diagram illustrating an implementation of a data export method according to a second embodiment of the present invention. As shown in fig. 3, taking two slices as an example, the specific implementation process of the data export method is as follows:

1. after receiving the export instruction, the export tool sends an acquisition request to a global transaction version manager (namely a global transaction version management node);

2. the global transaction version manager receives the acquisition request, and responds to the previous global transaction version number (TS1, namely the target global transaction version number) to the outbound tool;

3. the export tool creates a query task for each fragment according to the number of the fragments of the distributed database;

4. the export tool divides the query task into a plurality of query batches (namely, into at least one sub-query task) according to the distribution condition of the data in the table;

5. the export tool divides the batch to establish the inquiry command, send the inquiry request (namely inquiry command) carrying the version number information of the present global affair (namely/. STS! TS1 /) to each fragment (namely goal fragment);

6. each segment responds to the derived tool for the queried data result set (i.e., query result);

7. the export tool processes the data result set and writes the data into a file in a formatted mode (the formatted mode can be used for exporting the data according to a uniform data storage format);

8. and repeating the operations of the steps 5, 6 and 7 until the whole data table is completely traversed and completely written into the corresponding export file.

Wherein the export tool: the export tool may be a distributed database export tool, and is used for being responsible for the export function of the database and outputting the data of each database fragment to a corresponding export file in a formatted manner. Export tools may be deployed on export nodes.

Global transaction version manager: may refer to a management node responsible for maintaining the distributed global transaction version number.

Fragmenting: the data fragmentation can be referred to as database fragmentation, and data stored in the same database table is stored on a plurality of database hosts (namely, the database fragmentation) in a scattered manner through a certain specific condition, so that the effect of dispersing the load of a single host is achieved. A distributed database system may often have multiple partitions, and each partition often has a host and several standby machines, and the data stored by the host and the standby machines are consistent to achieve the backup effect.

Global Transaction version number (STS): may refer to a unique sequence of increments issued by the global transaction version manager to represent a snapshot of the distributed database at a time. The global transaction version number may consist of a physical time, which is a millisecond difference time from 00 minutes 00 seconds on day 00 of 01 month 01 1970, and a logical time, which is a sequence value that distinguishes different transactions at the same physical time.

Batch query: the query command can be segmented by index segmentation or paging, so as to perform full-table traversal on the data table in batches.

In this embodiment, by specifying the global transaction version number corresponding to the current time of a single fragment during export, it can be ensured that data of each export query of the single fragment is at the same time. When exporting, the exported data among a plurality of fragments can be ensured to be globally consistent at a certain moment by designating each fragment as the same global transaction version number. By the data export method provided by the embodiment, even if the distributed database is in an online service, the global consistency of the exported data can be ensured by the corresponding global transaction version number at a certain moment when the data is exported.

In one embodiment, the distributed database is out of service for external interfaces. When the distributed database is in an offline scene, that is, there is no online service, and the distributed database is in a global consistency state, it can be ensured that the data is consistent when being exported.

In one embodiment, an opportunity is selected for a lock table operation (which may be understood as an operation to lock the derived data table). Locking the tables is similar to an offline scenario, but only locking at the table level ensures that data at the time of export is consistent under the condition that a certain table is globally consistent.

EXAMPLE III

Fig. 4 is a schematic structural diagram of a data export apparatus according to a third embodiment of the present invention. As shown in fig. 4, the apparatus includes:

an obtaining module 310, configured to obtain a target global transaction version number when the export instruction is received;

a determining module 320, configured to determine at least one target segment according to the export instruction;

an executing module 330, configured to create a corresponding query task for the target segment based on the target global transaction version number, and perform a data export operation based on the query task.

In the data export apparatus provided in the third embodiment of the present invention, first, the obtaining module 310 obtains a target global transaction version number when receiving an export instruction; then, through the determining module 320, at least one target segment is determined according to the derivation instruction; finally, through the execution module 330, a corresponding query task is created for the target segment based on the target global transaction version number, and a data export operation is executed based on the query task. The device creates the query task containing the target global transaction version number for the determined target fragment based on the acquired target global transaction version number, and executes data export operation on the basis, so that the data corresponding to the target global transaction version number can be exported, the problem of inconsistent exported data in the online scene of the distributed database is solved, and the reliability of data export is improved.

Optionally, the obtaining module 310 includes:

a request sending unit, configured to send an acquisition request to a global transaction version management node, where the acquisition request is a request for acquiring the target global transaction version number;

and the version number receiving unit is used for receiving the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node according to the current global transaction version number.

Optionally, the determining module 320 includes:

the analysis unit is used for analyzing the export instruction to obtain the fragment where the data to be exported is located;

and the node determining unit is used for taking the segment where the data to be exported is located as a target segment, and the number of the target segments is at least one.

Optionally, the executing module 330 includes:

a task creating unit, configured to create, for the target segment, an inquiry task including the target global transaction version number;

the dividing unit is used for dividing the query task into at least one sub-query task according to the distribution information of the data to be exported on the target fragment;

and the sub-query tasks comprise the target global transaction version number and are executed in parallel.

Optionally, the executing module 33 further includes:

the command creating unit is used for creating a corresponding query command for each sub-query task in the query tasks and sending the query command to a corresponding target fragment, wherein the query command is a query statement command which is generated based on the sub-query tasks and contains the target global transaction version number;

the result receiving unit is used for receiving the query result returned by the target fragment;

and the processing unit is used for processing the query result and exporting a processing result.

Optionally, the query result includes target data to be exported, which is determined by the target segment according to the query command and corresponds to the target global transaction version number.

The data export device provided by the embodiment of the invention can execute the data export method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

Fig. 5 is a schematic structural diagram of a derived node according to a fourth embodiment of the present invention. The exporting node 10 is intended to represent various forms of digital computers such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The export node may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 5, the export node 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the export node 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the export node 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the exporting node 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the data derivation method.

In some embodiments, the data derivation method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed on the exporting node 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data derivation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data derivation method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here may be implemented on an export node having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the derived node. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

EXAMPLE five

Fig. 6 is a schematic structural diagram of a data export system according to a fifth embodiment of the present invention. As shown in fig. 6, the data export system includes: the export node 410 provided by the embodiment of the present invention, and at least one target segment 420;

export node 410 is connected to each target slice 420.

On the basis of the above embodiment, the data export system further includes:

and the global transaction version management node is configured to receive the acquisition request sent by the export node 410, determine a target global transaction version number according to the current global transaction version number, and return the target global transaction version number to the export node 410, where the acquisition request is a request for acquiring the target global transaction version number.

The global transaction version management node is connected to export node 410.

The global transaction version management node may be connected to each target segment 420 through a computing node in the distributed database, for example, the global transaction version management node is connected to a computing node in at least one distributed database, and the computing node is connected to each target segment 420, so as to implement indirect connection between the global transaction version management node and each target segment 420.

The data export system provided by the fifth embodiment can be used for executing the data export method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (11)

1. A data export method applied to an export node, the method comprising:

when an export instruction is received, acquiring a target global transaction version number;

determining at least one target fragment according to the export instruction;

and creating a corresponding query task for the target fragment based on the target global transaction version number, and executing data export operation based on the query task.

2. The method of claim 1, wherein obtaining the target global transaction version number comprises:

sending an acquisition request to a global transaction version management node, wherein the acquisition request is a request for acquiring the target global transaction version number;

and receiving the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node according to the current global transaction version number.

3. The method of claim 1, wherein the determining at least one target slice according to the export instruction comprises:

analyzing the export instruction to obtain the fragment where the data to be exported is;

and taking the fragments where the data to be exported are located as target fragments, wherein the number of the target fragments is at least one.

4. The method of claim 1, wherein creating the corresponding query task for the target segment based on the target global transaction version number comprises:

creating a query task containing the target global transaction version number for the target fragment;

dividing the query task into at least one sub-query task according to the distribution information of the data to be exported on the target fragment;

and the sub-query tasks comprise the target global transaction version number and are executed in parallel.

5. The method of claim 1, wherein performing a data derivation operation based on the query task comprises:

creating a corresponding query command for each sub-query task in the query task, and sending the query command to a corresponding target fragment, wherein the query command is a query statement command which is generated based on the sub-query task and contains the target global transaction version number;

receiving a query result returned by the target fragment;

and processing the query result and exporting a processing result.

6. The method of claim 5,

and the query result comprises target data to be exported, which is determined by the target fragment according to the query command and corresponds to the target global transaction version number.

7. A data derivation apparatus, comprising:

the obtaining module is used for obtaining a target global transaction version number when the export instruction is received;

a determining module, configured to determine at least one target segment according to the derivation instruction;

and the execution module is used for creating a corresponding query task for the target fragment based on the target global transaction version number and executing data export operation based on the query task.

8. A derivation node, the derivation node comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data derivation method of any of claims 1-6.

9. A computer-readable storage medium storing computer instructions for causing a processor to perform the data derivation method of any one of claims 1-6 when executed.

10. A data export system, the system comprising: an export node as claimed in claim 8, and at least one target shard.

11. The system of claim 10, further comprising:

and the global transaction version management node is used for receiving the acquisition request sent by the export node, determining a target global transaction version number according to the current global transaction version number, and returning the target global transaction version number to the export node, wherein the acquisition request is a request for acquiring the target global transaction version number.

Images