CN116541451A

CN116541451A - Data export method and device

Info

Publication number: CN116541451A
Application number: CN202310574249.2A
Authority: CN
Inventors: 徐磊
Original assignee: Advanced New Technologies Co Ltd
Current assignee: Advanced New Technologies Co Ltd
Priority date: 2019-02-02
Filing date: 2019-02-02
Publication date: 2023-08-04
Also published as: CN110008262B; CN110008262A

Abstract

One or more embodiments of the present disclosure provide a data export method and apparatus, where the method includes: firstly, acquiring a database snapshot at the query time by using a static cursor-based mode of a first thread, and then acquiring target data corresponding to each data identifier recorded in the database snapshot one by one from the database; introducing a data queue serving as a data cache pool, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

Description

Data export method and device

This patent application is application number: 201910106690.1, filing date: the invention relates to a data export method and a data export device, which are divisional applications of China patent application in 2019, 02 and 02.

Technical Field

The present document relates to the field of computers, and in particular, to a data export method and apparatus.

Background

At present, with the rapid development of internet technology, the data volume generated in the business processing process is increasing, so the data volume stored in the database is also increasing, and in each business system, there is a demand for exporting data from the database to an Excel file.

Currently, the process of exporting and generating Excel report by using data provided in the prior art specifically includes: firstly, all data are queried in a JVM heap memory in batches, and then the data are saved and an Excel file is generated. In this process, since all data is stored in the memory, when the data volume to be exported is larger than the price, the JVM memory cannot store so much data volume, which will cause the problem of OOM memory overflow, and will cause frequent GC recycling, which will seriously affect the performance and usability of the system.

Aiming at the problems, the prior art also provides a method for realizing data export and generating an Excel report by adopting a paging export mode, specifically, physical inquiry paging is carried out on data through SQL sentences, the data quantity of each inquiry is controlled, after the current paging data is processed, the current paging data is deleted and recycled, and then the next paging data is processed. In this process, multiple queries of data are required, and unique key ordering must exist for the SQL statement; therefore, in the case of data changes such as new addition, modification, deletion, etc. of data during export, the operations of searching and sorting pages multiple times will cause duplication and deletion of the exported data.

It follows that there is a need to provide a data export method that ensures data integrity and accuracy.

Disclosure of Invention

One or more embodiments of the present specification provide a data export method. The method comprises the following steps: and writing target data corresponding to the data identifications, which are sequentially acquired from the database, into a preset data queue according to each data identification in the database snapshot. And sequentially reading the target data from the preset data queue according to a first-in first-out sequence. And storing the target data into a pre-created SXSSF object so as to enable the target data to be transferred to a temporary file on a preset hard disk through the SXSSF object. And when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

One or more embodiments of the present specification provide a data export apparatus. The device comprises a data writing module, a data processing module and a data processing module, wherein the data writing module is used for writing target data corresponding to each data identifier in a database snapshot, which are sequentially acquired from the database, into a preset data queue. The device also comprises a data reading module which is used for sequentially reading the target data from the preset data queue according to a first-in first-out sequence. The device also comprises a file generation module which is used for storing the target data into a pre-created SXSSF object so as to enable the target data to be transferred into a temporary file on a preset hard disk through the SXSSF object. And when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

One or more embodiments of the present specification provide a data export device. The apparatus comprises a processor and a memory arranged to store computer executable instructions. And when the executable instruction is executed, the processor writes target data which is sequentially acquired from the database and corresponds to each data identifier in the database snapshot into a preset data queue. And sequentially reading the target data from the preset data queue according to a first-in first-out sequence. And storing the target data into a pre-created SXSSF object so as to enable the target data to be transferred to a temporary file on a preset hard disk through the SXSSF object. And when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

One or more embodiments of the present specification provide a storage medium. The storage medium is for storing computer-executable instructions. And when the executable instruction is executed by the processor, writing target data which is sequentially acquired from the database and corresponds to the data identifier into a preset data queue when each data identifier in the database snapshot. And sequentially reading the target data from the preset data queue according to a first-in first-out sequence. And storing the target data into a pre-created SXSSF object so as to enable the target data to be transferred to a temporary file on a preset hard disk through the SXSSF object. And when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

Drawings

For a clearer description of one or more embodiments of the present description or of the solutions of the prior art, the drawings that are necessary for the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are only some of the embodiments described in one or more of the present description, from which other drawings can be obtained, without inventive faculty, for a person skilled in the art.

FIG. 1 is a flow diagram of a method of data export provided in one or more embodiments of the present disclosure;

FIG. 2 is a schematic diagram of a first implementation of a data export method according to one or more embodiments of the present disclosure;

FIG. 3 is a schematic diagram illustrating a second implementation of a data derivation method according to one or more embodiments of the present disclosure;

FIG. 4 is a schematic diagram of a third implementation of a data export method according to one or more embodiments of the present disclosure;

FIG. 5 is a schematic diagram of a fourth implementation of a data export method according to one or more embodiments of the present disclosure;

FIG. 6 is a schematic diagram of a fifth implementation of a data export method according to one or more embodiments of the present disclosure;

FIG. 7 is a schematic diagram illustrating a module composition of a data deriving device according to one or more embodiments of the present disclosure;

fig. 8 is a schematic structural diagram of a data export device according to one or more embodiments of the present disclosure.

Detailed Description

In order for those skilled in the art to better understand the solutions in one or more embodiments of the present specification, the solutions in one or more embodiments of the present specification will be clearly and completely described below with reference to the drawings in one or more embodiments of the present specification, and it is apparent that the described embodiments are only a part of one or more embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one or more persons of ordinary skill in the art without undue burden from the disclosure, are intended to be within the scope of one or more of the disclosure.

One or more embodiments of the present disclosure provide a data export method and apparatus, which not only can implement streaming export of target data, avoid the problem of memory overflow caused by one-time export of data, but also can avoid the problem of data duplication and missing caused by data change in a paging export manner, thereby improving the data integrity and accuracy in the process of exporting mass data.

Fig. 1 is a flow chart of a data export method according to one or more embodiments of the present disclosure, where an execution subject of the method in fig. 1 is a computer device, and as shown in fig. 1, the method at least includes the following steps:

s101, acquiring a static cursor data set of a database by using a first thread, wherein the static cursor data set comprises: database snapshot at a specified time;

specifically, the static cursor data set refers to a database snapshot obtained based on a static cursor mode, wherein the static cursor is a cursor displaying a result set in a current state at the moment of opening the cursor, and the static cursor does not reflect updating, deleting or inserting of basic data when the cursor is opened; the static cursor stores the data set in the tempdb when opening, so that the displayed data is consistent with the data set when the cursor is opened, and the update of the database after the cursor is opened is not displayed in the cursor, for example, a JDBC cursor, which refers to a data result set returned by the JDBC query operation and capable of processing one line or a part of lines;

in addition, the specified time may be a query time, and the database snapshot records identification information (i.e., data identifier) of the line set data stored in the database at the query time, and the database snapshot does not change along with the change of the data stored in the database, so that the data acquired from the database based on a plurality of data identifiers in the database snapshot does not have the situation of data repetition and data missing;

S102, sequentially acquiring target data corresponding to each data identifier in the database snapshot by using a first thread from the database;

the database snapshot records data identifiers corresponding to a plurality of line set data stored in the database at the query time, and sequentially queries target data corresponding to each data identifier in the database; specifically, a vernier data set is obtained by adopting a JDBC vernier mode, data reading is carried out by adopting the vernier data set mode, all line set objects do not need to be returned at one time, a large amount of data does not need to be piled in a JVM memory, and data can be exported in a streaming export mode;

s103, sequentially writing the acquired target data into a preset data queue by using a first thread, so that the target data are sequentially read from the preset data queue by using a second thread according to a first-in first-out sequence, and a data export file is generated based on the read target data;

wherein the first thread may be referred to as a producer thread and the second thread may be referred to as a consumer thread, the first thread and the second thread being two mutually independent threads executing in parallel; specifically, after the first thread acquires the corresponding target data from the database, the target data are sequentially enqueued, namely stored in a preset data queue, so that the second thread can sequentially read the target data from the preset data queue;

The SQL statement is executed by an independent producer thread, and a read-only static cursor mode is adopted in JDBC from a database to read the database snapshot at the time of inquiry, wherein session link cannot be interrupted during data reading. Acquiring target data from a database one by one based on the database snapshot, and sequentially storing the target data in a preset data queue; simultaneously, the consumer thread reads target data from the preset data queue;

specifically, the producer thread obtains the target data corresponding to the data identifier from the database one by one based on the data identifier recorded in the database snapshot, and enqueues the target data one by one, the consumer thread reads the target data from the preset data queue one by one, i.e. dequeues the target data one by one, the consumer thread generates a data export file (e.g. an excel file) based on the read target data, and the target data is read in a streaming export mode, so that the read target data can be transferred to the hard disk, and the memory overflow phenomenon can be avoided when the excel file is generated.

In one or more embodiments of the present disclosure, a first thread is utilized to obtain a database snapshot at a query time based on a static cursor mode, and then target data corresponding to each data identifier recorded in the database snapshot is obtained from the database one by one; introducing a data queue serving as a data cache pool to transfer data, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

The preset data queue may be a bounded queue, a blocking queue, or a bounded blocking queue, where the bounded blocking queue may not only limit the maximum upper limit of the number of data stored in the queue, but also automatically implement data read-write blocking, and preferably, the bounded blocking queue is used as the preset data queue.

Wherein the bounded blocking queue is a first-in-first-out (FIFO) data structure and can accommodate a queue having an upper limit on the amount of data; and the bounded blocking queue is a queue supporting two additional operations, one additional operation being: reading a blocking operation, namely when the queue is empty, waiting for the queue to become non-empty by a consumer thread for acquiring the element; another additional operation is: a write blocking operation, i.e., when the queue is full, the producer thread storing the element will wait for the queue to be available, i.e., the bounded blocking queue stores the element for the producer thread and the consumer thread reads the element's container.

In specific implementation, as shown in fig. 2, a schematic diagram of a first implementation principle of a data export method is provided, where a process of exporting target data from a database to generate a data export file specifically includes:

S201, the producer thread acquires a static cursor data set of a database, wherein the static cursor data set comprises: database snapshot at a specified time;

s202, the producer thread sequentially acquires corresponding target data from a database based on a static cursor data set;

s203, the producer thread sequentially writes the acquired target data into a bounded blocking queue;

s204, the consumer thread sequentially reads target data from the bounded blocking queue according to a first-in first-out sequence;

s205, the consumer thread generates a data export file based on the read target data, for example, translates the target data and stores the translated target data in the sheet in the excel file.

In order to notify the consumer thread after the producer thread completes the data writing operation, so as to avoid the consumer thread from continuously waiting for reading the data, after the step S103 of sequentially writing the acquired target data into the preset data queue by using the first thread, the method further includes:

judging whether target data corresponding to all data identifiers in the database snapshot are acquired or not, specifically, if the first thread acquires the target data corresponding to all the data identifiers in the database snapshot one by one, writing the acquired target data into a preset data queue in sequence, and then, indicating that the operation of the producer thread data is finished;

If yes, end indication information aiming at the database snapshot is generated, and the end indication information is written into a preset data queue so as to transmit a data export end signal to a consumer thread;

if not, continuing to execute the step of sequentially acquiring the target data corresponding to each data identifier in the database snapshot by using the first thread from the database in the step of S102.

Correspondingly, after the first thread writes the end indication information into the preset data queue, the second thread will read the end indication information, so that the second thread is informed when the first thread ends data enqueuing, the second thread is prevented from continuously waiting for data reading operation, and based on the end indication information, if the reading result of the second thread is the end indication information, the second thread is controlled to stop executing the target data reading operation, and the data export operation is determined to be completed.

In specific implementation, as shown in fig. 3, a schematic diagram of a second implementation principle of a data export method is provided, where a process of exporting target data from a database to generate a data export file specifically includes:

s301, a producer thread acquires a static cursor data set of a database, wherein the static cursor data set comprises: database snapshot at a specified time;

S302, the producer thread sequentially acquires corresponding target data from a database based on a static cursor data set;

s303, the producer thread sequentially writes the acquired target data into a bounded blocking queue;

s304, after the producer thread processes the last line of data in the static cursor data set, writing an ending indication message into the bounded blocking queue; after target data corresponding to each data identifier in the database snapshot is obtained, end indication information is generated, and the end indication information is written into a preset data queue so as to inform a consumer of the end of data enqueuing operation, and the data is not required to be read continuously;

s305, the consumer thread sequentially reads target data from the bounded blocking queue according to a first-in first-out sequence;

s306, the consumer thread translates and saves the read target data to a corresponding storage location, for example, translates the target data and stores the translated target data in the sheet in the excel file;

s307, after the consumer thread reads the end indication information, the consumer thread knows that the producer thread has written all data into the queue, and generates a data export file based on the read target data.

The method comprises the steps that for target data written into a bounded blocking queue by a first thread, before target data is read from the bounded blocking queue by a second thread, the target data is in a non-free state, and when the target data is changed into a free state after the second thread executes a reading operation on the target data, a Java Virtual Machine (JVM) automatically carries out GC recycling processing on the target data in the free state, so that occupation of a memory is timely relieved, and after the target data is sequentially read from a preset data queue according to a first-in first-out sequence by the second thread, the method further comprises the following steps:

performing GC recycling treatment on target data in a free state in a preset data queue by using an automatic garbage recycling mechanism based on java; specifically, in the process of reading data from the database and storing the data into the memory, the Java virtual machine JVM performs GC garbage collection in real time, so that memory overflow in the process of reading the data can be avoided.

For the case that the preset data queue is a bounded blocking queue, the first thread is allowed to write data into the bounded blocking queue only when the number of data stored in the bounded blocking queue is smaller than the maximum storage number, otherwise, the first thread blocks waiting for writing operation, based on which S103 sequentially writes the acquired target data into the preset data queue by using the first thread, specifically including:

Judging whether the number of the target data stored in the bounded-blocking queue reaches a preset number upper limit, specifically, determining whether the bounded-blocking queue is in a full state or not based on the number of the target data stored in the bounded-blocking queue, and further determining whether to allow the first thread to write data into the bounded-blocking queue or not;

if the judgment result is negative, the first thread is utilized to sequentially write the acquired target data into the bounded blocking queue, specifically, the number of the target data stored in the bounded blocking queue does not reach the preset upper limit, the bounded blocking queue is not full, and the first thread is allowed to enqueue the target data at the moment.

If the judgment result is yes, executing a write-in blocking operation on the first thread, so that the first thread is blocked and waits for writing target data into the bounded blocking queue until the quantity of the target data stored in the bounded blocking queue is smaller than the preset quantity upper limit;

specifically, when the number of target data stored in the bounded blocking queue reaches the preset number upper limit, the bounded blocking queue is full, and the first thread is suspended to write data into the bounded blocking queue;

even though the Java virtual machine JVM automatically performs GC recycling processing on target data in a free state, when the writing speed of the first thread is greater than the reading speed of the second thread, the situation that the number of target data stored in the bounded blocking queue reaches the preset upper limit of the number may occur, and at this time, a new writing operation is automatically blocked, that is, the first thread enters a writing blocking waiting state;

Specifically, the first thread continues to add elements (i.e., target data) at the tail of the queue, and if the queue is full, the first thread will write to the block and will not be activated until there is an element dequeue operation, i.e., waiting for the queue to have a free position if the queue is full and then activating the first thread to write elements to the queue, i.e., writing to wait if the queue is full, until there is an element dequeue operation signaling activation of the block wait thread (i.e., producer thread).

In specific implementation, as shown in fig. 4, a schematic diagram of a third implementation principle of a data export method is provided, where a process of exporting target data from a database to generate a data export file specifically includes:

s401, the producer thread acquires a static cursor data set of a database, wherein the static cursor data set comprises: database snapshot at a specified time;

s402, the producer thread sequentially acquires corresponding target data from a database based on a static cursor data set;

s403, the producer thread sequentially writes the obtained target data into a bounded blocking queue;

s404, when the queue is full, the producer thread enters a write-blocking waiting state until new target data is dequeued, and then the S403 is continuously executed;

S405, after the producer thread processes the last line of data in the static cursor data set, writing an ending indication message into the bounded blocking queue; after target data corresponding to each data identifier in the database snapshot is obtained, end indication information is generated, and the end indication information is written into a preset data queue so as to inform a consumer of the end of data enqueuing operation, and the data is not required to be read continuously;

s406, the consumer thread sequentially reads the target data from the bounded blocking queue according to the first-in first-out sequence;

s407, the consumer thread translates and saves the read target data to a corresponding storage location, for example, translates the target data and stores the translated target data in the sheet in the excel file;

s408, after the consumer thread reads the end indication information, the consumer thread knows that the producer thread has written all data into the queue, and generates a data export file based on the read target data.

Further, considering that when the second thread reading speed is greater than the first thread writing speed, there may be a situation that the bounded blocking queue becomes empty, that is, there is no target data for the second thread to read in the bounded blocking queue, based on this, if the reading result of the second thread is empty, a read blocking operation is performed on the second thread, so that the second thread blocks and waits for reading the target data from the bounded blocking queue until there is target data in a non-free state in the bounded blocking queue.

Specifically, when the bounded blocking queue is empty, suspending the second thread from reading data from the bounded blocking queue; the Java virtual machine JVM automatically carries out GC recycling processing on target data in a free state, so that when the reading speed of a second thread is higher than the writing speed of a first thread, the situation that a bounded blocking queue becomes empty possibly occurs, and new reading operation is automatically blocked at the moment, namely the second thread enters a reading blocking waiting state;

specifically, the second thread continues to read elements (i.e., target data) from the head of the queue, and if the queue is empty, the second thread will read the block, and the second thread will not be activated until there is an element enqueuing operation, i.e., waiting for the queue to store a new element if the queue is empty and then activating the second thread to read elements from the queue, i.e., reading the wait if the queue is empty, until there is an element enqueuing operation to signal activation of the block waiting thread (i.e., consumer thread).

In specific implementation, as shown in fig. 5, a schematic diagram of a fourth implementation principle of a data export method is provided, where a process of exporting target data from a database to generate a data export file specifically includes:

S501, a producer thread acquires a static cursor data set of a database, wherein the static cursor data set comprises: database snapshot at a specified time;

s502, the producer thread sequentially acquires corresponding target data from a database based on a static cursor data set;

s503, the producer thread sequentially writes the obtained target data into a bounded blocking queue;

s504, when the queue is full, the producer thread enters a write-blocking waiting state until new target data is dequeued, and then the process continues to S503;

s505, after the producer thread processes the last line of data in the static cursor data set, writing an ending indication message into the bounded blocking queue; after target data corresponding to each data identifier in the database snapshot is obtained, end indication information is generated, and the end indication information is written into a preset data queue so as to inform a consumer of the end of data enqueuing operation, and the data is not required to be read continuously;

s506, the consumer thread sequentially reads the target data from the bounded blocking queue according to the first-in first-out sequence;

s507, when the queue is empty, the consumer thread enters a read blocking waiting state until new target data is enqueued, and then the S506 is continuously executed;

S508, the consumer thread translates and saves the read target data to a corresponding storage location, for example, translates the target data and stores the translated target data in the sheet in the excel file;

s509, after the consumer thread reads the end indication information, the consumer thread knows that the producer thread has written all data into the queue, and generates a data export file based on the read target data.

In view of the fact that the SXSSF object has the characteristic of automatically transferring data, in order to avoid the problem of memory overflow during the process of generating the data export file and achieve the purpose of simplifying the transferring data, the SXSSF object is introduced as a storage unit in the memory for storing data, based on which the data export file is generated by using the second thread based on the read target data, specifically including:

step one, the read target data is stored into a pre-established SXSSF object by utilizing a second thread, so that the target data is transferred into a temporary file on a preset hard disk through the SXSSF object;

specifically, a preset number of SXSSF objects can be created in a JVM memory in advance based on an apache POI, namely, a preset number of SXSSFWorkbook implementation classes are built, when a large amount of data is exported, the second thread stores the read target data into the SXSSF objects, and when the target data are stored in the preset number of SXSSF objects, the SXSSF objects can automatically transfer the target data from the JVM memory to the hard disk, namely, the POI is directly written into an xml temporary file on the hard disk, so that the subsequent second thread can continuously store the target data read from the bounded blocking queue into the SXSSF objects;

The producer thread puts the target data obtained from the database into a bounded blocking queue, the consumer thread takes the target data out of the bounded blocking queue and stores the target data in an SXSSF object, and the SXSSF can transfer a part of target data into a hard disk when the memory is insufficient;

step two, when the data export operation is determined to be completed, loading target data from a temporary file on a preset hard disk by using a second thread and generating a data export file;

when the second thread reads the ending indication information from the bounded blocking queue, the second thread determines that the data export operation is completed, and then the second thread generates a data export file based on the read target data.

Specifically, when target data is stored in a preset number of SXSSF objects, the SXSSF objects can automatically transfer the target data from the JVM memory to the hard disk, so that the second thread can continuously store the target data read from the bounded blocking queue into the SXSSF objects; wherein, a part of target data is not transferred to the hard disk, but is stored in the SXSSF object on the memory of the JVM finally; therefore, for the generation process of the data export file, the second thread is utilized to generate the data export file based on the target data stored in the JVM memory and the target data in the temporary file transferred to the preset hard disk, so that the memory space is not occupied because a part of the target data is transferred to the hard disk, the problem that the memory overflows due to overlarge exported data in the process of generating the data export file can be avoided, and meanwhile, the SXSSF object has the characteristic of automatically transferring the data, and the purpose of simplifying the transferred data can be achieved.

It should be noted that, because the SXSSF object has an automatic dump property, the second thread is preferably used to store the read target data into the SXSSF object, which has an effect of simplifying the dump process, and in addition, other ways may be used to implement data dump.

In specific implementation, as shown in fig. 6, a schematic diagram of a fifth implementation principle of a data export method is provided, where a process of exporting target data from a database to generate a data export file specifically includes:

s601, a producer thread acquires a static cursor data set of a database, wherein the static cursor data set comprises: database snapshot at a specified time;

s602, the producer thread sequentially acquires corresponding target data from a database based on a static cursor data set;

s603, the producer thread sequentially writes the acquired target data into a bounded blocking queue;

s604, when the queue is full, the producer thread enters a write-blocking waiting state until new target data is dequeued, and then the S603 is continuously executed;

s605, after the producer thread processes the last line of data in the static cursor data set, writing end indication information into the bounded blocking queue; after target data corresponding to each data identifier in the database snapshot is obtained, end indication information is generated, and the end indication information is written into a preset data queue so as to inform a consumer of the end of data enqueuing operation, and the data is not required to be read continuously;

S606, the consumer thread sequentially reads the target data from the bounded blocking queue according to the first-in first-out sequence;

s607, when the queue is empty, the consumer thread enters a read blocking waiting state until new target data is enqueued, and then continues to execute S606;

s608, the consumer thread stores the read target data into an SXSSF object so as to enable the target data to be transferred into a temporary file on a hard disk through the SXSSF object;

s609, after the consumer thread reads the end indication information, the consumer thread knows that the producer thread has written all data into the queue, loads target data from the temporary file on the hard disk and generates a data export file.

According to the data export method in one or more embodiments of the present disclosure, first, a database snapshot at a query time is obtained by using a static cursor-based manner of a first thread, and then target data corresponding to each data identifier recorded in the database snapshot is obtained from the database one by one; introducing a data queue serving as a data cache pool, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

Corresponding to the data export method described in fig. 1 to 6, based on the same technical concept, one or more embodiments of the present disclosure further provide a data export device, and fig. 7 is a schematic block diagram of the data export device provided in one or more embodiments of the present disclosure, where the device is configured to perform the data export method described in fig. 1 to 6, and as shown in fig. 7, the device includes:

a data acquisition module 701, configured to acquire a static cursor data set of a database by using a first thread, where the static cursor data set includes: database snapshot at a specified time; sequentially acquiring target data corresponding to each data identifier in the database snapshot from the database;

a data writing module 702, configured to sequentially write the target data into a preset data queue by using the first thread;

a data reading module 703, configured to sequentially read the target data from the preset data queue by using a second thread according to a first-in first-out order;

a file generating module 704, configured to generate a data export file based on the read target data by using the second thread.

Optionally, the preset data queue includes: a bounded queue, a blocked queue, or any one of a bounded blocked queue.

Optionally, the apparatus further comprises: an end indication writing module for:

after the target data are written into a preset data queue in sequence, judging whether the target data corresponding to each data identifier in the database snapshot are acquired or not;

if yes, end indication information aiming at the database snapshot is generated, and the end indication information is written into the preset data queue.

Optionally, the apparatus further comprises: a first read operation control module for:

and if the reading result of the second thread is the ending indication information, controlling the second thread to stop executing the target data reading operation, and determining that the data export operation is finished.

Optionally, if the preset data queue is a bounded blocking queue; the data writing module 702 is specifically configured to:

judging whether the number of the target data stored in the bounded blocking queue reaches a preset upper number limit;

and if the judgment result is negative, writing the target data into the bounded blocking queue in sequence.

Optionally, the data writing module 702 is further specifically configured to:

if the judgment result is yes, executing the write-in blocking operation on the first thread, so that the first thread is blocked and waits for writing target data into the bounded blocking queue until the number of the target data stored in the bounded blocking queue is smaller than a preset upper limit.

Optionally, the apparatus further comprises: a second read operation control module for:

and if the reading result of the second thread is null, executing a reading blocking operation on the second thread to enable the second thread to block and wait for reading target data from the bounded blocking queue until the bounded blocking queue has target data in a non-free state.

Optionally, the apparatus further comprises: the data recovery processing module is used for:

after the target data are written into a preset data queue in sequence, performing GC recycling processing on the target data in a free state in the preset data queue based on a java automatic garbage recycling mechanism.

Optionally, the file generating module 704 is specifically configured to:

storing the target data into a pre-created SXSSF object by using the second thread so as to enable the target data to be transferred into a temporary file on a preset hard disk through the SXSSF object;

And when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

The data deriving device in one or more embodiments of the present disclosure firstly obtains a database snapshot at a query time by using a static cursor-based manner of a first thread, and then obtains target data corresponding to each data identifier recorded in the database snapshot from the database one by one; introducing a data queue serving as a data cache pool, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

It should be noted that, in the present specification, the embodiment about the data deriving device and the embodiment about the data deriving method in the present specification are based on the same inventive concept, so the specific implementation of this embodiment may refer to the implementation of the corresponding data deriving method, and the repetition is not repeated.

Further, according to the method shown in fig. 1 to 6, based on the same technical concept, one or more embodiments of the present disclosure further provide a data export apparatus for performing the data export method as shown in fig. 8.

The data export device may vary considerably in configuration or performance and may include one or more processors 801 and memory 802, where the memory 802 may store one or more stored applications or data. Wherein the memory 802 may be transient storage or persistent storage. The application programs stored in memory 802 may include one or more modules (not shown in the figures), each of which may include a series of computer-executable instructions in the data export device. Still further, the processor 801 may be arranged to communicate with a memory 802 to execute a series of computer executable instructions in the memory 802 on a data derivation device. The data-export device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input-output interfaces 805, one or more keyboards 806, and the like.

In a particular embodiment, a data export device includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the data export device, and configured to be executed by one or more processors, the one or more programs comprising computer-executable instructions for:

obtaining a static cursor data set of a database by using a first thread, wherein the static cursor data set comprises: database snapshot at a specified time;

for each data identifier in the database snapshot, sequentially acquiring target data corresponding to the data identifier from the database;

and sequentially writing the target data into a preset data queue, so that the target data is sequentially read from the preset data queue by a second thread according to a first-in first-out sequence, and a data export file is generated based on the read target data.

Optionally, the preset data queue comprises: a bounded queue, a blocked queue, or any one of a bounded blocked queue.

Optionally, the computer executable instructions, when executed, further comprise, after sequentially writing the target data into a preset data queue:

judging whether target data corresponding to each data identifier in the database snapshot is acquired or not;

Optionally, when the computer executable instruction is executed, if the read result of the second thread is the end indication information, the second thread is controlled to stop executing the target data read operation, and it is determined that the current data export operation is completed.

Optionally, when the computer executable instructions are executed, if the preset data queue is a bounded blocking queue; the writing the target data into a preset data queue in turn comprises:

Optionally, when the computer executable instruction is executed, if the result of the determination is yes, performing a write blocking operation on the first thread, so that the first thread blocks and waits for writing target data into the bounded-blocking queue until the number of target data stored in the bounded-blocking queue is less than a preset upper number limit.

Optionally, when the computer executable instruction is executed, if the read result of the second thread is null, a read blocking operation is performed on the second thread, so that the second thread blocks and waits for reading target data from the bounded blocking queue until the bounded blocking queue has target data in a non-free state.

and performing GC recycling treatment on the target data in the free state in the preset data queue by using an automatic garbage recycling mechanism based on java.

Optionally, the computer executable instructions, when executed, generate a data export file based on the read target data, comprising:

The data export device in one or more embodiments of the present disclosure firstly obtains a database snapshot at a query time by using a static cursor-based manner of a first thread, and then obtains target data corresponding to each data identifier recorded in the database snapshot from the database one by one; introducing a data queue serving as a data cache pool, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

It should be noted that, the embodiments related to the data exporting apparatus in the present specification and the embodiments related to the data exporting method in the present specification are based on the same inventive concept, so the specific implementation of this embodiment may refer to the implementation of the foregoing corresponding data exporting method, and the repetition is omitted.

Further, according to the method shown in fig. 1 to 6, based on the same technical concept, one or more embodiments of the present disclosure further provide a storage medium, which is used to store computer executable instructions, and in a specific embodiment, the storage medium may be a U disc, an optical disc, a hard disk, etc., where the computer executable instructions stored in the storage medium can implement the following flow when executed by a processor:

Optionally, the storage medium stores computer executable instructions that, when executed by the processor, the preset data queue comprises: a bounded queue, a blocked queue, or any one of a bounded blocked queue.

Optionally, the computer executable instructions stored on the storage medium, when executed by the processor, further comprise, after sequentially writing the target data into a preset data queue:

Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, if the read result of the second thread is the end indication information, the second thread is controlled to stop executing the target data read operation, and it is determined that the current data export operation is completed.

Optionally, the computer executable instructions stored by the storage medium, when executed by the processor, if the predetermined data queue is a bounded blocking queue; the writing the target data into a preset data queue in turn comprises:

Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, if the result of the determination is yes, performing a write blocking operation on the first thread, so that the first thread blocks and waits for writing target data into the bounded blocking queue until the number of the target data stored in the bounded blocking queue is less than a preset upper number limit.

Optionally, when executed by the processor, the computer executable instructions stored in the storage medium perform a read blocking operation on the second thread if the read result of the second thread is null, so that the second thread blocks waiting to read target data from the bounded-blocking queue until the bounded-blocking queue has target data in a non-free state.

Optionally, the computer executable instructions stored on the storage medium, when executed by the processor, generate the data-derived file based on the read target data, including:

When the computer executable instructions stored in the storage medium in one or more embodiments of the present disclosure are executed by the processor, firstly, a database snapshot at a query time is obtained by using a first thread based on a static cursor mode, and then target data corresponding to each data identifier recorded in the database snapshot is obtained from the database one by one; introducing a data queue serving as a data cache pool, and storing the acquired target data into the data queue by using a first thread; then, the second thread is utilized to read target data from the data queue one by one, and a corresponding data export file is generated based on the read target data, so that the streaming export of the target data can be realized, the problem of memory overflow caused by one-time export of the data is avoided, the problem of data repetition and deletion caused by changing the data in a paging export mode is avoided, and the data integrity and accuracy of the mass data export process are improved.

It should be noted that, in the present specification, the embodiment about the storage medium and the embodiment about the data deriving method in the present specification are based on the same inventive concept, so the specific implementation of this embodiment may refer to the implementation of the corresponding data deriving method, and the repetition is omitted.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but also HDL is not only one, but a plurality of, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HD Cal, JHDL (Java Hardware Description Language), lava, lola, my HDL, palam, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when one or more of the present description are implemented.

One skilled in the relevant art will recognize that one or more of the embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more of the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

One or more of the present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

One or more of the present description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is merely illustrative of one or more embodiments of the present disclosure and is not intended to limit the one or more embodiments of the present disclosure. Various modifications and alterations to one or more of this description will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of one or more of the present description, are intended to be included within the scope of the claims of one or more of the present description.

Claims

1. A data export method comprising:

for each data identifier in the database snapshot, writing target data which are sequentially acquired from the database and correspond to the data identifier into a preset data queue;

sequentially reading the target data from the preset data queue according to a first-in first-out sequence;

storing the target data into a pre-created SXSSF object so that the target data is transferred to a temporary file on a preset hard disk through the SXSSF object;

2. The method according to claim 1, wherein for each data identifier in the database snapshot, writing the target data corresponding to the data identifier sequentially acquired from the database into a preset data queue, includes: the method comprises the steps that a first thread is utilized for writing target data corresponding to data identifications sequentially acquired from a database into a preset data queue aiming at each data identification in a database snapshot;

the reading the target data from the preset data queue sequentially according to a first-in first-out sequence comprises the following steps: sequentially reading the target data from the preset data queue according to a first-in first-out sequence by using a second thread;

The storing the target data into the pre-created SXSSF object comprises: and storing the target data into a pre-created SXSSF object by utilizing the second linearity.

3. The method of claim 1, wherein the preset data queue comprises: a bounded queue, a blocked queue, or any one of a bounded blocked queue.

4. The method according to claim 2, wherein after writing target data corresponding to the data identifier sequentially acquired from the database into a preset data queue, further comprising:

5. The method of claim 4, further comprising:

6. The method of claim 2, wherein if the predetermined data queue is a bounded blocking queue; the writing the target data corresponding to the data identifier, which are sequentially acquired from the database, into a preset data queue includes:

if the judgment result is negative, writing the target data corresponding to the data identifier, which are sequentially acquired from the database, into a preset data queue.

7. The method of claim 6, further comprising:

8. The method of claim 6, further comprising:

9. The method of claim 1, wherein after writing target data corresponding to the data identifier sequentially acquired from the database into a preset data queue, further comprising:

10. A data export device comprising:

the data writing module is used for writing target data corresponding to each data identifier in the database snapshot into a preset data queue, wherein the target data are sequentially acquired from the database and correspond to the data identifiers;

the data reading module is used for sequentially reading the target data from the preset data queue according to a first-in first-out sequence;

the file generation module is used for storing the target data into a pre-created SXSSF object so as to enable the target data to be transferred into a temporary file on a preset hard disk through the SXSSF object; and when the data export operation is determined to be completed, loading the target data from the temporary file and generating a data export file.

11. The device according to claim 10,

the data writing module is used for writing target data corresponding to each data identifier in the database snapshot into a preset data queue, wherein the target data are sequentially acquired from the database and correspond to the data identifiers by utilizing a first thread;

the data reading module is used for sequentially reading the target data from the preset data queue by using a second thread according to a first-in first-out sequence;

And the file generation module is used for storing the target data into a pre-created SXSSF object by utilizing the second linearity.

12. The apparatus of claim 10, wherein the preset data queue comprises: a bounded queue, a blocked queue, or any one of a bounded blocked queue.

13. The apparatus of claim 11, wherein the apparatus further comprises: an end indication writing module for:

after target data corresponding to the data identifiers sequentially acquired from the database are written into a preset data queue, judging whether the target data corresponding to each data identifier in the database snapshot is acquired or not;

14. The apparatus of claim 13, wherein the apparatus further comprises: a first read operation control module for:

15. The apparatus of claim 11, wherein if the predetermined data queue is a bounded blocking queue; the data writing module is specifically configured to:

16. The apparatus of claim 15, wherein the data writing module is further specifically configured to:

17. The apparatus of claim 15, wherein the apparatus further comprises: a second read operation control module for:

18. The apparatus of claim 10, wherein the apparatus further comprises: the data recovery processing module is used for:

After target data corresponding to the data identifier, which are sequentially acquired from the database, are written into a preset data queue, performing GC recycling processing on the target data in a free state in the preset data queue based on a java automatic garbage recycling mechanism.

19. A data export device comprising:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

20. A storage medium storing computer executable instructions that when executed by a processor implement the following: