CN112100245B - Data export method, device, electronic equipment and storage medium - Google Patents
Data export method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112100245B CN112100245B CN202010990293.8A CN202010990293A CN112100245B CN 112100245 B CN112100245 B CN 112100245B CN 202010990293 A CN202010990293 A CN 202010990293A CN 112100245 B CN112100245 B CN 112100245B
- Authority
- CN
- China
- Prior art keywords
- data
- export
- database
- pagenum
- acquiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 230000001174 ascending effect Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Abstract
The embodiment of the invention relates to the field of communication and discloses a data export method, a device, electronic equipment and a storage medium. The data deriving method comprises the following steps: acquiring the quantity F of export files which need to be generated when data export is carried out on a database; acquiring the available thread number P of the hardware equipment for carrying out the data export work; and executing at least two data export tasks on the database in parallel by adopting the P available threads, and generating F export files based on data exported by executing the data export tasks on the database. And the method is applied to mongoDB, and quick data export is realized.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a data export method, a data export device, electronic equipment and a storage medium.
Background
MongoDB is an open source database system written in the C++ language based on distributed file storage. In the prior art, when data export is required for MongoDB, firstly, the number of new files required to be generated for data export is obtained, then, the data is exported from MongoDB, and each new file is sequentially generated in a queue mode. In particular, the method of generating each new file may include: firstly, a paging query command is adopted to query data from MongoDB, then the queried data is assembled into result data according to a self-defined rule, and finally, the result data is written into a corresponding new file.
However, in implementing embodiments of the present invention, the inventors found that: when the data quantity to be exported in the MongoDB is large, the method for exporting data from the MongoDB in the prior art and sequentially generating each new file in a queue mode is low in efficiency, and the data export speed is low.
Disclosure of Invention
An object of an embodiment of the present invention is to provide a data export method, apparatus, electronic device, and storage medium, capable of improving data export speed.
In order to solve the above technical problems, an embodiment of the present invention provides a data export method, including: acquiring the quantity F of export files which need to be generated when data export is carried out on a database; acquiring the available thread number P of the hardware equipment for carrying out the data export work; executing at least two data export tasks on the database in parallel by adopting the P available threads, and generating F export files based on data exported by executing the data export tasks on the database; wherein executing at least two data export tasks on the database in parallel using the P available threads, comprises: when a single available thread is adopted to execute a single data export task on the database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of the preset data storage amount is exported from the queried total amount of data according to the sequence of the query time from front to back and used as data exported by executing the single data export task.
The embodiment of the invention also provides a data export device, which comprises:
The first acquisition module is used for acquiring the quantity F of the export files which are needed to be generated when the data export is carried out on the database;
the second acquisition module is used for acquiring the number P of available threads of the hardware equipment for carrying out the data export work;
The generation module is used for executing at least two data export tasks on the database in parallel by adopting the P available threads acquired by the second acquisition module, and generating F export files based on data exported by executing the data export tasks on the database; wherein executing at least two data export tasks on the database in parallel using the P available threads, comprises: when a single available thread is adopted to execute a single data export task on the database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of the preset data storage amount is exported from the queried total amount of data according to the sequence of the query time from front to back and used as data exported by executing the single data export task.
The embodiment of the invention also provides electronic equipment, which comprises:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data export method described above.
An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the data export method described above.
Compared with the prior art, the method and the device can acquire the available thread number P of the hardware device for data export work, conduct data export on the database through the P available threads simultaneously and in parallel, and for F exported files needing to be completed, conduct operation through the P available threads in parallel circulation, so that the data export speed is greatly improved, and the problem that the data export work speed of each exported file is slow when the data export work of each exported file needs to be completed from the database in sequence in a queue mode in the prior art is solved.
In addition, in the data export method provided in the embodiment of the present invention, in the method for executing at least two data export tasks on the database in parallel using the P available threads, F export file clocks are generated based on data of executing data processing tasks on the database, and the method for generating each export file includes: and generating each export file according to the primary key id of the database and the data pageData contained in a single page in the database.
In addition, in the data export method provided by the embodiment of the present invention, the generating each export file according to the primary key id of the database and the data pageData included in a single page in the database includes: acquiring a paging page number pageNum of the initial data in the export file in the database; performing id ascending sorting on the database, and acquiring data pageData to be exported from pageNum after id ascending sorting according to a paging query method, wherein id is a serial number of data in the database; acquiring the number foundCount of the queried data to be exported according to pageData; if foundCount is greater than or equal to the total number of lines L of the export file, acquiring target export data from the pageData 1; and generating the export file according to the target export data. Compared with the prior art, the embodiment of the invention only needs to acquire the paging page number pageNum of the initial data in the export file in the database and perform id ascending sequencing on the database, namely the data pageData to be exported in the pageNum can be acquired through the paging query method of the database, and complex service filtering conditions are not required to be set; in addition, as only the data pageData to be exported in the pageNum is queried, the amount of query data is very small, and the data query speed is improved; and, only need according to the relation of the number foundCount of the data to be exported that has been inquired and L, can judge whether the operation to export the data is all finished, if foundCount is greater than or equal to L, reveal pageData include all L line data, obtain the goal export data from pageData1, produce the said export file according to the goal export data, the logic of the data export method is simple, need not set up the complicated logic operation.
In addition, in the data export method provided in the embodiment of the present invention, after the number foundCount of the queried data to be exported is obtained according to pageData, before the export file is generated according to the target export data, the method further includes:
If foundCount is smaller than L, acquiring the serial number ID of the last line of data in the PageNum; taking the ID larger than the ID as a query condition, and acquiring data pageData to be exported from the pagenum+1 according to the conditional query method; updating the pagenum=pagenum+1; updating the foundCount according to the pageData and pageData2; if the updated foundCount is greater than or equal to the L, obtaining target derived data from the pageData and pageData2; and if the updated foundCount is smaller than the L, circularly executing the step of acquiring the serial number ID of the last line of data in the pageNum, taking the ID larger than the ID as a query condition, acquiring the data pageData to be exported from the pagenum+1 according to the paging query method, updating the pagenum=pagenum+1, and updating the step of foundCount according to the pageData1 and pageData. Compared with the prior art, if foundCount is smaller than L, the subsequent query can acquire the data pageData to be exported from the pagenum+1 according to the paging query method by taking ID larger than ID as a query condition, and because ID is a serial number of the data in the database and is a main key of the database, the data can be rapidly positioned to the starting position of the current paging query by taking the condition that ID is larger than ID, compared with the prior method which needs to scan a large amount of invalid data, the data export speed is further improved by the technical scheme provided by the embodiment. In addition, according to the technical scheme provided by the embodiment of the invention, as long as the updated foundCount is smaller than L, the data contained in the subsequent paging page number in the database can be queried according to the paging query method of the database by taking the ID larger than the ID as the query condition, and the query condition is simple and easy to realize.
In addition, in the data export device provided in this embodiment, the generating module is composed of P generating sub-modules, and each generating sub-module is specifically configured to generate each export file according to the primary key id of the database and the data pageData included in a single page in the database.
In addition, in the data deriving device provided in this embodiment, the generating submodule includes:
a first obtaining unit, configured to obtain a pageNum of a paging page number of the initial data in the export file in the database;
The second obtaining unit is configured to perform id ascending order on the database, and obtain data pageData to be exported from pageNum after id ascending order according to a paging query method, where id is a serial number of data in the database;
A third obtaining unit, configured to obtain, according to the pageData1 obtained by the second obtaining unit, the number foundCount of queried data to be exported;
A fourth obtaining unit configured to obtain target export data from the pageData1 if foundCount obtained by the third obtaining unit is equal to or greater than the total number of rows L of the export file;
And the generation unit is used for generating the export file according to the target export data acquired by the third acquisition unit.
In addition, in the data export device provided in this embodiment, the generating sub-module further includes:
a fifth acquiring unit, configured to acquire a serial number ID of the last line of data in the pageNum acquired by the first acquiring unit, if the foundCount acquired by the third acquiring unit is smaller than the L;
A sixth obtaining unit, configured to obtain, according to the condition query method, data pageData2 to be exported from pagenum+1 with the ID that is greater than the ID obtained by the fifth obtaining unit as a query condition;
a first updating unit configured to update pagenum=pagenum+1 acquired by the first acquiring unit;
a second updating unit configured to update the foundCount acquired by the third acquiring unit according to the pageData1 acquired by the second acquiring unit and the pageData2 acquired by the sixth acquiring unit;
The fourth obtaining unit is further configured to obtain target derived data from the pageData1 obtained by the second obtaining unit and the pageData2 obtained by the sixth obtaining unit if foundCount updated by the second updating unit is equal to or greater than the L;
And the circulating unit is used for circularly executing the fifth acquiring unit, the sixth acquiring unit, the first updating unit and the second updating unit if foundCount updated by the second updating unit is smaller than L.
Drawings
One or more embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which the figures of the drawings are not to be taken in a limiting sense, unless otherwise indicated.
FIG. 1 is a flow chart of a data export method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data export method according to a second embodiment of the present invention;
FIG. 3 is a flowchart I of a method for generating each exported file in the data exporting method according to the second embodiment of the present invention shown in FIG. 2;
FIG. 4 is a second flowchart of a method for generating each exported file in the data exporting method according to the second embodiment of the present invention shown in FIG. 2;
Fig. 5 is a schematic structural diagram of a data deriving device according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of the second acquisition module 502 in the data deriving device according to the third embodiment of the present invention shown in fig. 5;
Fig. 7 is a schematic diagram of a structure of a generating sub-module 5031 in the data deriving device according to the third embodiment of the present invention shown in fig. 5;
fig. 8 is a second schematic structural diagram of a generating sub-module 5031 in the data deriving device according to the third embodiment of the present invention shown in fig. 5;
Fig. 9 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings. However, those of ordinary skill in the art will understand that in various embodiments of the present application, numerous technical details have been set forth in order to provide a better understanding of the present application. The claimed application may be practiced without these specific details and with various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not be construed as limiting the specific implementation of the present application, and the embodiments can be mutually combined and referred to without contradiction.
A first embodiment of the present invention relates to a data export method, the flow of which is shown in fig. 1, including:
step 101, obtaining the number F of export files needed to be generated when data export is performed on the database.
Step 102, obtaining the number P of available threads of the hardware device for data export work.
Step 103, executing at least two data export tasks on the database in parallel by using P available threads, and generating F export files based on data exported by executing the data export tasks on the database;
wherein, adopt P available thread to carry out at least two data export tasks to the database in parallel, include:
When a single available thread is adopted to execute a single data export task on a database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of a preset data storage amount is exported from the queried total amount of data according to the sequence of the query time from front to back and used as data exported by executing the single data export task.
Compared with the prior art, the method and the device can acquire the available thread number P of the hardware device for data export work, conduct data export on the database through the P available threads simultaneously and in parallel, and for F exported files needing to be completed, conduct operation through the P available threads in parallel circulation, so that the data export speed is greatly improved, and the problem that the data export work speed of each exported file is slow when the data export work of each exported file needs to be completed from the database in sequence in a queue mode in the prior art is solved.
A second embodiment of the present invention relates to a data export method, the flow of which is shown in fig. 2, including:
Step 201, obtaining the number F of export files to be generated when data export is performed on the database.
Specifically, step 201 may first obtain the number of lines L included in each export file, where in this embodiment, the number of lines included in each export file is the same, and all lines are L; and then, acquiring the total line number N of the data in the database, and rounding up the result of the calculation of N/L to obtain F.
Further, in this embodiment, L may be not only specified directly, but also obtained through empirical conversion according to a preset export file size.
Step 202, the available CPU core number P1 of the hardware device for data export is obtained.
In this embodiment, the available CPU core number P1 of the hardware device may be the actual core number of the CPU, and in order to ensure that other services on the hardware device can be performed normally, P1 may also be a part of the actual core number of the CPU, which may be specifically obtained by the following formula (1):
P1＝n*p (1)
Where p is the actual number of cores of the CPU, n is the available coefficient, n e (0, 1), for example n=0.6 in this embodiment.
Step 203, according to F and P1, the number of threads P of the hardware device performing the data export job is obtained.
In this embodiment, step 203 may specifically take the smaller one of the values F and P1 as P, i.e.: if F > P1, p=p1, if F < P1, p=f. In this embodiment, F > P1 and p=p1 will be described as an example.
In step 204, at least two data export tasks are executed on the database in parallel by using the P available threads, and F export files are generated based on the data exported by executing the data export tasks on the database.
In this embodiment, step 203 may perform F tasks in parallel loop through P available threads, specifically, a fixed-length thread pool may be created according to P, and in a Java program, may be created through ExecutorService executor =executors.
Specifically, in this embodiment, step 204 may generate each export file according to the primary key id of the database and the data pageData included in a single page in the database.
Further, in this embodiment, the F derived files generated in step 204 may be specifically T 1,T2,……,TF, and in order to enable those skilled in the art to more clearly understand the method of generating each derived file in step 204, taking the generation of derived file T 2 as an example, as shown in fig. 3, the method includes the following steps:
Step 301, obtaining the paging page number pageNum of the initial data in the export file in the database.
In this embodiment, step 301 may specifically obtain pageNum by the following procedure:
int pageNum＝(int)((taskSeq-1)*L/pageSize+1)
Wherein int represents an integer; taskSeq is the number of the export file, in this embodiment, the export file is T 2, so taskSeq =2; l is the number of lines of T 2; pageSize is the number of data lines per page in the database.
Step 302, performing id ascending order on the database, and obtaining data pageData to be exported from pageNum after id ascending order according to the paging query method, wherein id is the serial number of the data in the database.
In this embodiment, step 302 may specifically complete the id ascending order by the following procedure (2):
query.with(new Sort(new Order(Direction.ASC,"_id"))) (2)
step 302 may specifically complete the paging query by the following procedure (3), obtain pageData1:
query.skip((pageNum-1)*pageSize).limit(pageSize) (3)
Skip ((pageNum-1) × pageSize) represents skip (pageNum-1) × pageSize rows of data in the database, limit (pageSize) represents read pageSize rows of data in the database.
As can be seen from the above procedure (3), pageData is specifically pageSize lines of data that are continuously read after skipping (pageNum-1) pageSize lines of data in the database.
Step 303, obtaining the number foundCount of the queried data to be exported according to pageData 1.
In this embodiment foundCount is an initial value of 0. After pageData a1 is acquired at step 302, foundCount = pageSize.
Step 304, if foundCount is greater than or equal to L, obtain target export data from pageData.
Specifically, first, a serial number ID star of the T 2 start data in a database is obtained, where ID star = (taskSeq-1) x l+1; then, obtaining a serial number ID end of the data of the T 2 th item in the database, wherein ID end = taskSeq ×l; finally, target derived data is obtained from pageData a by the following procedure (4):
pageData＝ pageData1.subList(IDstar,IDend) (4)
Wherein pageData is the target export data.
Step 305, generating an export file from the target export data.
In this embodiment, the specific manner of generating the export file is not limited, and in the actual use process, step 305 may operate on the target export data by any custom manner to generate the export file, which is not described herein in detail.
In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present invention, the following specific descriptions will be given by taking the derived file as T 2, l=5, and pagesize=17 as an example, where the steps shown in fig. 3 are as follows:
first, the pageNum is obtained by step 301:
int pageNum＝(int)((taskSeq-1)*L/pageSize+1)
int pageNum＝(int)((2-1)*5/17+1)
int pageNum＝(int)(1.29)
pageNum＝1
next, pageData1 is acquired through step 302:
query.skip((pageNum-1)*pageSize).limit(pageSize)
query.skip((1-1)*17).limit(17)
query.skip(0).limit(17)
namely: pageData1 are 17 data in the first page in the database.
Third, through step 303, foundCount is obtained
In this embodiment, since pageData is 17 pieces of data in the first page in the database, foundCount =17.
Fourth, since foundCount =17, l=5, foundCount > L, the target derived data pageData is obtained according to step 304pageData1, specifically, ID star＝(taskSeq-1)*L+1＝(2-1)*5+1＝6,IDend = taskSeq ×l= 2*5 =10, at this time, pageData =pagedata1.sublist (IDstar, ID) =pagedata1.sublist (6, 10), that is, pageData is the first 6-10 lines of data paged in the database.
Fifth, an export file is obtained according to step 305. Wherein T 2 contains the first paged 6-10 rows of data in the database.
Compared with the prior art, the technical scheme shown in fig. 3 only needs to acquire the paging page number pageNum of the initial data in the export file in the database and perform id ascending sorting on the database, namely, the data pageData to be exported in the pageNum can be acquired through the paging query method of the database, and complex service filtering conditions are not required to be set; in addition, as only the data pageData to be exported in the pageNum is queried, the amount of query data is very small, and the data query speed is improved; and, only need according to the relation of the number foundCount of the data to be exported that has been inquired and L, can judge whether the operation to export the data is all finished, if foundCount is greater than or equal to L, reveal pageData include all L line data, obtain the goal export data from pageData1, produce the said export file according to the goal export data, the logic of the data export method is simple, need not set up the complicated logic operation.
Further, as shown in fig. 4, after step 303, before step 305, the method may further include:
Step 306, if foundCount < L, obtaining the serial number ID of the last line of data in the PageNum.
In step 307, the data pageData to be exported is obtained from the pagenum+1 according to the conditional query method with ID > ID as the query condition.
Specifically, step 307 may obtain pageData2 by the following procedure (5)
find({"_id":{$gt:ID}}).limit(pageSize) (5)
As can be seen from the above procedure (5), pageData is pageSize lines of data contained in PageNum+1.
Step 308, update pageNum = pageNum +1.
Step 309, update foundCount according to pageData and pageData.
Specifically, step 309 may be obtained foundCount by the following formula (6):
foundCount＝ pageData1+ pageData2-L％pageSize (6)
＝pageSize+pageSize-L％pageSize
＝2*pageSize-L％pageSize
where L% pageSize represents the remainder of dividing L by pageSize.
Step 310, if foundCount is greater than or equal to L after updating, target export data is obtained from pageData and pageData.
Specifically, first, a serial number ID star of the T 2 start data in a database is obtained, where ID star = (taskSeq-1) x l+1; then pageData 1=pagedata1. Sublist (ID star, ID); thirdly, obtaining the truncated sequence number endIndex = pageSize- (foundCount-L) of T 2 in pagenum+1; fourth, pageData2 = pagedata2.Sublist (0, endindex); fifth, target derived data pageData = pageData1+ pageData2.
Step 311, if foundCount < L after updating, loop execution steps 306-309.
In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present invention, the following specific description will be given by taking the derived file as T 2, l=14, and pagesize=6 as an example, where the steps shown in fig. 4 are as follows:
first, the pageNum is obtained by step 301:
int pageNum＝(int)((taskSeq-1)*L/pageSize+1)
int pageNum＝(int)((2-1)*14/6+1)
int pageNum＝(int)(3.33)
pageNum＝3
next, pageData1 is acquired through step 302:
query.skip((pageNum-1)*pageSize).limit(pageSize)
query.skip((3-1)*6).limit(6)
query.skip(12).limit(6)
namely: pageData1 are 6 data in the third page in the database.
Third, foundCount =6 is acquired by step 303.
Fourth, since foundCount =6 and l=14, oundCount < L, according to step 306, the sequence number id=18 of the last line of data in pagenum=3 is obtained.
Fifthly, according to step 307, using ID > ID as query condition, according to the paging query method, obtaining the data pageData2 to be exported from pagenum+1: find ({ "_id": { $gt:18 }). Limit (6).
Sixth, pagenum=pagenum+1=3+1=4 is updated according to step 308.
Seventh, foundCount is updated according to step 309.
foundCount＝2*pageSize-L％pageSize＝2*6-14％6＝12-2＝10
Eighth, since foundCount =10 and l=12 after updating, foundCount < L, obtaining the serial number id=24 of the last line of data in pagenum=4 through step 306; according to step 307, using ID > ID as query condition, obtaining data pageData' to be exported from pageNum+1 according to paging query method: find ({ "_id": { $gt:24 }). Limit (6); updating pagenum=pagenum+1=4+1=5 according to step 308; foundCount = 3*6-14% 6=18-2=16 is updated according to step 309.
Ninth, since foundCount =16 and l=12 after updating, foundCount > L, according to step 310:
1. Obtaining a serial number ID star of the T 2 start data in the database, wherein ID star = (taskSeq-1) l+1=15;
2、pageData1＝pageData1.subList(15，18)；
3. obtain pageData2 = find ({ "_id": { $gt:18 }). Limit (6);
4. Fetch T 2 truncated sequence number endIndex = pageSize- (foundCount-L) =6- (16-14) =4 in the fifth page in the database;
5、pageData2’＝pageData2’.subList(0,endIndex)＝pageData2’.subList(0,4)；
6. Target derived data pageData = pageData1+ pageData + pageData2'.
Tenth, an export file is obtained according to step 305. Wherein T 2 contains the third paged 3-6 rows of data and the fourth paged pageSize rows of data in the database; and the fifth page of 1-4 rows of data.
Compared with the prior art, if foundCount is smaller than L, the subsequent query can acquire the data pageData to be exported from the pagenum+1 according to the paging query method by taking ID larger than ID as the query condition, and the ID is the serial number of the data in the database and is the main key of the database, so that the data export speed can be further improved by rapidly locating the beginning position of the current paging query under the condition that ID is larger than ID, compared with the conventional method which needs to scan a large amount of invalid data, the technical scheme provided by the embodiment queries valid data. In addition, according to the technical scheme provided by the embodiment of the invention, as long as the updated foundCount is smaller than L, the data contained in the subsequent paging page number in the database can be queried according to the paging query method of the database by taking the ID larger than the ID as the query condition, and the query condition is simple and easy to realize.
A third embodiment of the present invention relates to a data deriving device, as shown in fig. 5, including:
A first obtaining module 501, configured to obtain the number F of export files that need to be generated when exporting data from a database;
A second obtaining module 502, configured to obtain the number P of threads available to the hardware device for performing the data export work;
A generating module 503, configured to execute at least two data export tasks on the database in parallel using the P available threads acquired by the second acquiring module, and generate F export files based on data exported by executing a data export task on the database; wherein executing at least two data export tasks on the database in parallel using the P available threads, comprises: when a single available thread is adopted to execute a single data export task on the database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of the preset data storage amount is exported from the queried total amount of data according to the sequence of the query time from front to back and used as data exported by executing the single data export task.
Further, as shown in fig. 6, the second obtaining module 502 includes:
A first obtaining submodule 601, configured to obtain an available CPU core number P1 of the hardware device;
and a second obtaining sub-module 602, configured to obtain the P according to the F and the P1 obtained by the first obtaining sub-module 601.
Further, the generating module 503 is configured to include a generating sub-module 5031, and each generating sub-module 5301 is specifically configured to generate each export file according to a primary key id of the database and data pageData included in a single page in the database.
Further, as shown in fig. 7, the generating submodule 5031 includes:
A first obtaining unit 701, configured to obtain a pageNum of a paging page number of the start data in the export file in the database;
A second obtaining unit 702, configured to perform id ascending order on the database, and obtain, according to a paging query method, data pageData1 to be exported from pageNum after id ascending order, where id is a serial number of data in the database;
a third obtaining unit 703, configured to obtain, according to the pageData1 obtained by the second obtaining unit 702, the number foundCount of queried data to be exported;
A fourth obtaining unit 704, configured to obtain target export data from the pageData1 if foundCount obtained by the third obtaining unit 703 is equal to or greater than the total number of rows L of the export file;
a generating unit 705 configured to generate the export file according to the target export data acquired by the fourth acquiring unit 704.
Further, as shown in fig. 8, the generating submodule 5031 further includes:
A fifth acquiring unit 706, configured to acquire a serial number ID of the last line of data in the pageNum acquired by the first acquiring unit 701, if the foundCount acquired by the third acquiring unit 703 is smaller than the L;
A sixth obtaining unit 707, configured to obtain, according to a conditional query method, data pageData to be exported from pagenum+1 with the ID that is greater than the ID obtained by the fifth obtaining unit 706 as a query condition;
a first updating unit 708 configured to update pagenum=pagenum+1 acquired by the first acquiring unit 701;
a second updating unit 709 for updating the foundCount acquired by the third acquiring unit 703 according to the pageData1 acquired by the second acquiring unit 702 and the pageData2 acquired by the sixth acquiring unit 707;
the fourth obtaining unit 704 is further configured to obtain target derived data from the pageData1 obtained by the second obtaining unit 702 and the pageData2 obtained by the sixth obtaining unit 707 if foundCount updated by the second updating unit 709 is greater than or equal to the L;
a loop unit 710 configured to loop the fifth acquiring unit 706, the sixth acquiring unit 707, the first updating unit 708, and the second updating unit 709 if foundCount updated by the second updating unit 709 is smaller than the L.
The specific implementation method of the data export device provided by the embodiment of the present invention may be referred to the data export method provided by the embodiment of the present invention, and will not be described herein.
A fourth embodiment of the present invention relates to an electronic apparatus, as shown in fig. 9, including:
At least one processor 901; and
A memory 902 communicatively coupled to the at least one processor 901; wherein,
The memory 902 stores instructions executable by the at least one processor 901 to enable the at least one processor 901 to perform the data export methods according to the first and second embodiments of the present invention.
Where the memory and the processor are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses connecting the various circuits of the one or more processors and the memory together. The bus may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or may be a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over the wireless medium via the antenna, which further receives the data and transmits the data to the processor.
The processor is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory may be used to store data used by the processor in performing operations.
A fifth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program implements the above-described method embodiments when executed by a processor.
That is, it will be understood by those skilled in the art that all or part of the steps in implementing the methods of the embodiments described above may be implemented by a program stored in a storage medium, where the program includes several instructions for causing a device (which may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps in the methods of the embodiments of the application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples of carrying out the invention and that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Claims (6)
1. A data export method, comprising:
Acquiring the quantity F of export files which need to be generated when data export is carried out on a database;
Acquiring the available CPU core number P1 of the hardware equipment for carrying out the data export work, and acquiring the thread number P of the hardware equipment for carrying out the data export work according to the F and the P1;
Executing at least two data export tasks on the database in parallel by adopting the P available threads, and generating F export files based on data exported by executing the data export tasks on the database;
Wherein executing at least two data export tasks on the database in parallel using the P available threads, comprises:
When a single available thread is adopted to execute a single data export task on the database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of a preset data storage amount is exported from the queried total amount of data according to the sequence of query time from front to back and used as data exported by executing the single data export task;
In executing at least two data export tasks on the database in parallel by using the P available threads, generating F export files based on data of executing data processing tasks on the database, and generating each export file, the method comprises the following steps:
generating each export file according to the primary key id of the database and the data pageData contained in a single page in the database; the generating each export file according to the primary key id of the database and the data pageData included in the single page in the database includes:
Acquiring a paging page number pageNum of the initial data in the export file in the database;
Performing id ascending sorting on the database, and acquiring data pageData to be exported from pageNum after id ascending sorting according to a paging query method, wherein id is a serial number of data in the database;
acquiring the number foundCount of the queried data to be exported according to pageData;
If foundCount is greater than or equal to the total number of lines L of the export file, acquiring target export data from the pageData 1;
and generating the export file according to the target export data.
2. The method of claim 1, wherein after the obtaining the number of queried data to be exported foundCount according to the pageData a, before the generating the exported file according to the target exported data, further comprises:
if foundCount is smaller than L, acquiring the serial number ID of the last line of data in the PageNum;
taking the ID larger than the ID as a query condition, and acquiring data pageData to be exported from the pagenum+1 according to a conditional query method;
Updating the pagenum=pagenum+1;
updating the foundCount according to the pageData and pageData 2;
if the updated foundCount is greater than or equal to the L, obtaining target derived data from the pageData and pageData 2;
And if the updated foundCount is smaller than the L, circularly executing the step of acquiring the serial number ID of the last line of data in the pageNum, taking the ID larger than the ID as a query condition, acquiring the data pageData to be exported from the pagenum+1 according to the paging query method, updating the pagenum=pagenum+1, and updating the step of foundCount according to the pageData1 and pageData.
3. A data deriving apparatus, comprising:
The first acquisition module is used for acquiring the quantity F of the export files which are needed to be generated when the data export is carried out on the database;
The second acquisition module is used for acquiring the available CPU core number P1 of the hardware equipment for carrying out the data export work and acquiring the thread number P of the hardware equipment for carrying out the data export work according to the F and the P1;
The generation module is used for executing at least two data export tasks on the database in parallel by adopting the P available threads acquired by the second acquisition module, and generating F export files based on data exported by executing the data export tasks on the database; wherein executing at least two data export tasks on the database in parallel using the P available threads, comprises: when a single available thread is adopted to execute a single data export task on the database, if the total amount of data queried by executing the single data export task is larger than the maximum storage amount of data of a single export file, data of a preset data storage amount is exported from the queried total amount of data according to the sequence of query time from front to back and used as data exported by executing the single data export task;
The generation module consists of P generation sub-modules, and each generation sub-module is specifically configured to generate each export file according to a primary key id carried by the database and data pageData contained in a single page in the database;
The generating submodule includes:
a first obtaining unit, configured to obtain a pageNum of a paging page number of the initial data in the export file in the database;
The second obtaining unit is configured to perform id ascending order on the database, and obtain data pageData to be exported from pageNum after id ascending order according to a paging query method, where id is a serial number of data in the database;
A third obtaining unit, configured to obtain, according to the pageData1 obtained by the second obtaining unit, the number foundCount of queried data to be exported;
A fourth obtaining unit configured to obtain target export data from the pageData1 if foundCount obtained by the third obtaining unit is equal to or greater than the total number of rows L of the export file;
And the generation unit is used for generating the export file according to the target export data acquired by the third acquisition unit.
4. The apparatus of claim 3, wherein the generating sub-module further comprises:
a fifth acquiring unit, configured to acquire a serial number ID of the last line of data in the pageNum acquired by the first acquiring unit, if the foundCount acquired by the third acquiring unit is smaller than the L;
A sixth obtaining unit, configured to obtain, according to a conditional query method, data pageData to be exported from pagenum+1 with the ID that is greater than the ID obtained by the fifth obtaining unit as a query condition;
A first updating unit configured to update pagenum=pagenum+1 acquired by the first acquiring unit;
a second updating unit configured to update the foundCount acquired by the third acquiring unit according to the pageData1 acquired by the second acquiring unit and the pageData2 acquired by the sixth acquiring unit;
The fourth obtaining unit is further configured to obtain target derived data from the pageData1 obtained by the second obtaining unit and the pageData2 obtained by the sixth obtaining unit if foundCount updated by the second updating unit is equal to or greater than the L;
And the circulating unit is used for circularly executing the fifth acquiring unit, the sixth acquiring unit, the first updating unit and the second updating unit if foundCount updated by the second updating unit is smaller than L.
5. An electronic device, comprising:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data export method according to claim 1 or 2.
6. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the data export method of claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010990293.8A CN112100245B (en) | 2020-09-18 | Data export method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010990293.8A CN112100245B (en) | 2020-09-18 | Data export method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112100245A CN112100245A (en) | 2020-12-18 |
| CN112100245B true CN112100245B (en) | 2024-05-31 |
Family
ID=
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996067A (en) * | 2009-08-19 | 2011-03-30 | 阿里巴巴集团控股有限公司 | Data export method and device |
| WO2017005094A1 (en) * | 2015-07-03 | 2017-01-12 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN107656968A (en) * | 2017-08-31 | 2018-02-02 | 武汉斗鱼网络科技有限公司 | High-volume business datum deriving method and system |
| CN108628890A (en) * | 2017-03-21 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of data export method and system |
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996067A (en) * | 2009-08-19 | 2011-03-30 | 阿里巴巴集团控股有限公司 | Data export method and device |
| WO2017005094A1 (en) * | 2015-07-03 | 2017-01-12 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN108628890A (en) * | 2017-03-21 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of data export method and system |
| CN107656968A (en) * | 2017-08-31 | 2018-02-02 | 武汉斗鱼网络科技有限公司 | High-volume business datum deriving method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| DE60004640T2 (en) | METHOD AND DEVICE FOR JUMP FORECASTING USING A HYBRID BRANCH HISTORY WITH A COMMON ACCESS STRUCTURE | |
| CN104021460A (en) | Work flow management system and work flow handling method | |
| CN104750727B (en) | A kind of column memory storage inquiry unit and column memory storage querying method | |
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| US7698312B2 (en) | Performing recursive database operations | |
| CN112860692B (en) | Database table structure conversion method and device and electronic equipment thereof | |
| CN111738590A (en) | Production scheduling method and device | |
| CN105279269A (en) | SQL generating method and system for supporting table free association | |
| CN112580319A (en) | Data processing method, device, equipment and computer readable storage medium | |
| CN104268050A (en) | Simple method for testing bandwidth of internal memory | |
| CN112100245B (en) | Data export method, device, electronic equipment and storage medium | |
| DE102018125971A1 (en) | SYSTEMS AND METHOD FOR CALCULATING HALBBYTES SCALA PRODUCTS IN OPERANDS FROM TWO TILES | |
| CN116955538B (en) | Medical dictionary data matching method and device, electronic equipment and storage medium | |
| CN109857740B (en) | Character string storage method, matching method, electronic device and readable storage medium | |
| CN112100245A (en) | Data export method and device, electronic equipment and storage medium | |
| CN115543317A (en) | Front-end page development method and device | |
| CN110737671A (en) | Table-based retrieval method and device | |
| CN111414452A (en) | Search word matching method and device, electronic equipment and readable storage medium | |
| US20040034443A1 (en) | Numerical controller | |
| CN112667744B (en) | Method and device for synchronously updating data in database in batch | |
| CN110597615B (en) | Method for processing coding instruction and node equipment | |
| CN110825444B (en) | Pin configuration method of electronic control unit | |
| US11409523B2 (en) | Graphics processing unit | |
| CN112506944B (en) | Data standard conversion access method, device, equipment and medium between service systems | |
| CN109978081B (en) | Method, apparatus, device and medium for determining feature transformation mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |