CN113609226A - Data export method and device, computer equipment and storage medium - Google Patents
Data export method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113609226A CN113609226A CN202110911398.4A CN202110911398A CN113609226A CN 113609226 A CN113609226 A CN 113609226A CN 202110911398 A CN202110911398 A CN 202110911398A CN 113609226 A CN113609226 A CN 113609226A
- Authority
- CN
- China
- Prior art keywords
- data
- scanning
- thread task
- result data
- scanning result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012216 screening Methods 0.000 claims abstract description 53
- 238000013467 fragmentation Methods 0.000 claims description 26
- 238000006062 fragmentation reaction Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 12
- 238000009795 derivation Methods 0.000 claims description 9
- 230000011218 segmentation Effects 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000007405 data analysis Methods 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 239000012634 fragment Substances 0.000 description 4
- 230000001174 ascending effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011284 combination treatment Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention relates to the field of big data, and discloses a data export method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a data export request, wherein the data export request carries screening conditions; creating a main thread task according to the screening condition, scanning data in the specified database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in a preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to the first storage device at the cloud end through the sub-thread task, the downloading link of the target file is generated, and the downloading link is written into the second storage device at the local part, so that the data exporting efficiency is improved. The present invention relates to blockchain techniques, such as data can be written into blockchains for use in scenarios such as data forensics.
Description
Technical Field
The present invention relates to the field of big data, and in particular, to a data export method and apparatus, a computer device, and a storage medium.
Background
With the increasing development of big data, there are often scenes of exporting data in business systems, and at present, data exporting tools are generally adopted to export data, but the following defects are easily existed when data exporting tools are used to export data: reading while writing can not be realized, the data export speed is slow, the export data occupies high memory when the data volume is high, and even the memory overflow can occur. Therefore, a data export scheme is needed to solve the above-mentioned drawbacks of data export.
Disclosure of Invention
Embodiments of the present invention provide a data export method, apparatus, computer device, and storage medium, which can perform data fragmentation processing to achieve read while write when exporting data, reduce memory resource consumption, separate read from statistics, and increase data processing speed, thereby improving data export efficiency.
In a first aspect, an embodiment of the present invention provides a data export method, including:
acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval;
creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download.
Further, the creating a main thread task according to the screening condition, and scanning data in an assigned database by the main thread task to obtain scanning result data includes:
dividing the time between the export start time and the export end time into equal segments according to the segment time interval carried in the screening condition;
creating a main thread task according to each time period after the equal division and fragmentation between the export start time and the export end time;
and scanning the data in the specified database in each time period through the main thread task to obtain the scanning result data.
Further, the scanning, by the main thread task, data in the specified database at each time period to obtain the scanning result data includes:
when no data is scanned in the designated database in the first time period after the equal division and fragmentation, merging the second time period and the third time period after the equal division and fragmentation;
scanning the data of the designated database in the time period after the merging processing of the second time period and the third time period;
and when data are scanned in the time periods after the merging, combining every two adjacent time periods after the second time period and the third time period, and scanning the data of the specified database according to the time periods after the combining to obtain the scanned data.
Further, the method further comprises:
when no data is scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period after the equally dividing and slicing;
scanning the data of the designated database in 4 adjacent time periods after the third time period;
and when data are scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period, and scanning the data of the specified database according to the time periods after the merging processing to obtain the scanned data.
Further, after the data in the specified database is scanned by the main thread task to obtain scan result data, the method further includes:
acquiring the data volume of the scanning result data scanned from the specified database in each time period after the equal division and fragmentation;
and adding marks to the scanning result data in each time period according to the data volume of the scanning result data in each time period, wherein the marks of the scanning data in different time periods are different.
Further, the method further comprises:
when the scanning result data exist in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value;
when the fact that the data quantity of the scanning result data stored in the preset queue is larger than a first preset threshold value is monitored, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to suspend scanning of the data in the specified database;
and when the data volume of the scanning result data in the preset queue is smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of the segment of the equal segmentation, and scanning the data in the designated database according to the increased time interval of the segment of the equal segmentation.
Further, the method further comprises:
acquiring scanning progress information of scanning the data in the specified database through the main thread task;
and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In a second aspect, an embodiment of the present invention provides a data export apparatus, including:
the device comprises an acquisition unit, a data export unit and a data analysis unit, wherein the acquisition unit is used for acquiring a data export request, and the data export request carries screening conditions which comprise export start time, export end time and segment time interval;
the scanning unit is used for creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
the writing unit is used for creating a sub-thread task when the scanning result data exist in the preset queue, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
and the generating unit is used for transmitting the target file to a first storage device at the cloud end through the sub-thread task when the scanning is detected to be finished, generating a downloading link of the target file through the sub-thread task, and writing the downloading link into a local second storage device for a user to download.
In a third aspect, an embodiment of the present invention provides a computer device, including a processor and a memory, where the memory is used for storing a computer program, and the processor is configured to invoke the computer program to execute the method of the first aspect.
In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program, where the computer program is executed by a processor to implement the method of the first aspect.
The embodiment of the invention can obtain a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval; creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download. By the method, data can be processed in a fragmentation mode when the data is exported, reading and writing can be achieved, memory resource consumption is reduced, reading and statistics are separated, the data processing speed is higher, and therefore the efficiency of exporting the data is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a data export method provided by an embodiment of the present invention;
FIG. 2 is a schematic diagram of a time profile provided by an embodiment of the present invention;
FIG. 3 is a schematic block diagram of a data export apparatus according to an embodiment of the present invention;
fig. 4 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data export method provided by the embodiment of the invention can be applied to a data export device, and in some embodiments, the data export device is arranged in computer equipment. In certain embodiments, the computer device includes, but is not limited to, one or more of a smartphone, tablet, laptop, and the like.
The embodiment of the invention can obtain a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval; creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download.
The embodiment of the invention scans data from the designated database through the screening condition fragments, simultaneously stores the scanning result data into the preset queue, and writes the scanning result data acquired from the preset queue through the sub-thread task into the target file, thereby realizing reading while writing, reducing the consumption of memory resources and improving the efficiency of data derivation.
The data derivation method provided by the embodiment of the present invention is schematically described below with reference to fig. 1.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data export method according to an embodiment of the present invention, and as shown in fig. 1, the method may be executed by a data export apparatus, where the data export apparatus is disposed in a computer device. Specifically, the method of the embodiment of the present invention includes the following steps.
S101: acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval.
In the embodiment of the present invention, a data export device may obtain a data export request, where the data export request carries a screening condition, and the screening condition includes an export start time, an export end time, and a segment time interval.
In certain embodiments, the export start time and export end time in the screening condition are times before a current time; the segment time interval is a time-sliced time interval between the export start time and the export end time, and is used for equally dividing the time between the export start time and the export end time. In some embodiments, the derivation start time, the derivation end time, and the time interval of the segment are shown in fig. 2, and fig. 2 is a schematic diagram of a time distribution provided by an embodiment of the present invention.
In some embodiments, the filtering condition is used to indicate a time at which data needs to be scanned from the specified database and each time period after the time is equally divided to create the main thread task, so that the main thread task can scan the data in the specified database at each time period. In this way, it is facilitated to control the data amount and scanning speed of data in the scan specification database by controlling the size of each time period.
S102: and creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue.
In the embodiment of the present invention, the data export device may create a main thread task according to the screening condition, scan data in the designated database through the main thread task to obtain scan result data, and transmit the scan result data to a preset queue. In certain embodiments, the specified database includes, but is not limited to, a MySQL database.
In one embodiment, when the data export device creates a main thread task according to the screening condition and scans data in a specified database through the main thread task to obtain scan result data, the data export device may divide the time between the export start time and the export end time into equal pieces according to the time interval of the piece segment carried in the screening condition; creating a main thread task according to each time period after the equal division and fragmentation between the export start time and the export end time; and scanning the data in the specified database in each time period through the main thread task to obtain the scanning result data.
In an embodiment, when the data in the designated database is scanned by the main thread task in each time period to obtain the scan result data, the data exporting apparatus may perform merging processing on the second time period and the third time period after the equal division and fragmentation when no data is scanned in the designated database in the first time period after the equal division and fragmentation. In one embodiment, the rule for merging the time segments after fragmentation is according to 2nAnd selecting the number of time periods for combination treatment, wherein the value of n is 0, 1, 2 and 3.
In one embodiment, when the data export device scans the data in the specified database in the various time periods by the main thread task to obtain the scanning result data, the data export device divides the data into the first time period (i.e. 2)0) When no data is scanned in the designated database, the second time period and the third time period after the equal division and fragmentation are two adjacent time periods (2)1) Carrying out merging treatment; scanning the data of the designated database in the time period after the merging processing of the second time period and the third time period; and when data are scanned in the time periods after the merging, combining every two adjacent time periods after the second time period and the third time period, and scanning the data of the specified database according to the time periods after the combining to obtain the scanned data.
In one embodiment, the data export means divides the third equally sliced period by 4 (2) when no data is scanned during the combined period2) Merging of adjacent time segmentsC, processing; scanning the data of the designated database in 4 adjacent time periods after the third time period; and when data are scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period, and scanning the data of the specified database according to the time periods after the merging processing to obtain the scanned data.
In one embodiment, when no data is scanned in the time period after the merging process of 4 adjacent time periods after the third time period, 8 (i.e. 2) after the seventh time period may be used3) And merging the adjacent time periods, and scanning the data of the specified database according to each time period after merging.
In one embodiment, after the data export device scans data in a specified database through the main thread task to obtain scan result data, the data export device may obtain the data volume of the scan result data scanned from the specified database in each time period after the equal division and fragmentation; and adding marks to the scanning result data in each time period according to the data volume of the scanning result data in each time period, wherein the marks of the scanning data in different time periods are different. In some embodiments, the data amount of the scan result data may include, but is not limited to, the number of pages, the number of strips, and the like of the scan result data.
In certain embodiments, the identification may include, but is not limited to, a numeric identification, an alphabetic identification, and the like. In one embodiment, when the identifiers are added to the scan result data in the respective time periods, the identifiers may be added in an ascending order.
In one example, if the current page is 500, and the number of the scanning result data in each sliced time period is 500, an identifier is added to each piece of scanning result data in ascending order of numbers, assuming that when 500 pieces of scanning result data of the current page are searched in a scanning manner, the identifier of the 500 th piece is obtained to be maxld, the maximum maxld may be introduced into the next paging scanning, and when the data of the next page of the current page is scanned in the sliced time period, the identifier of the scanning result data of the next page is added in ascending order of identifiers larger than the maxld based on the maxld.
In one embodiment, the data exporting apparatus may obtain scanning progress information for scanning data in the specified database by the main thread task; and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device. In certain embodiments, the second storage may include, but is not limited to, a redis cache. Monitoring the progress of the scanning is facilitated by storing the scanning progress information in the second storage means.
In an embodiment, when the data export apparatus transmits the scanning result data to the preset queue, the data export apparatus may sequentially transmit the scanning result data to the preset queue according to the sequence of obtaining the scanning result data.
S103: and when the scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file.
In the embodiment of the present invention, the data export device may create a sub-thread task when it is detected that the scanning result data exists in the preset queue, acquire the scanning result data from the preset queue through the sub-thread task, and write the scanning result data into the target file.
In one embodiment, when the data export device obtains the scanning result data from the preset queue through the sub-thread task, the data export device may obtain the scanning result data from the preset queue in batch through the sub-thread task.
In one embodiment, when writing the scanning result data into a target file, the data export apparatus may create the target file locally through the sub-thread task, perform statistics on the scanning result data acquired from a preset queue, and write the counted scanning result data into the target file.
In an embodiment, when the sub-thread task does not obtain the scanning result data from the preset queue, the sub-thread task waits until the sub-thread task obtains the scanning result data from the preset queue, and the sub-thread task starts to obtain the scanning result data from the preset queue and writes the scanning result data into a target file.
In one embodiment, the data deriving device may monitor whether the data amount of the scan result data in the preset queue is greater than a first preset threshold when detecting that the scan result data exists in the preset queue; and when the fact that the data quantity of the scanning result data stored in the preset queue is larger than a first preset threshold value is monitored, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to suspend scanning of the data in the specified database. In some embodiments, the first preset threshold may be any preset value, such as 5000. In this way, it is avoided that the data reading speed is much higher than the writing speed, resulting in excessive data being accumulated in the queue.
In an embodiment, when it is monitored that the data volume of the scanning result data in the preset queue is smaller than a second preset threshold, a second notification message is sent to the main thread task, where the second notification message is used to notify the main thread task to increase a segment time interval of an equal segment, and scan the data in the specified database according to the increased segment time interval. In some embodiments, the second preset threshold may be any preset value, such as 2000. In this way, the reading speed can be prevented from being far lower than the writing speed, and the data reading speed can be improved.
In one embodiment, when the time interval of the segment of the equal segment is increased by the main thread task, the segment may be enlarged according to the original time interval of the segment according to a preset enlargement rule.
In one example, assuming the original fragmentation time interval is 5 minutes, the fragmentation time interval may be droppedUp to 2mAnd 5, the value of m is 0, 1, 2, 3. and m, and what value is specifically selected for m can be determined by sequentially selecting and according to whether the data volume of the data obtained by scanning at the amplified segment time interval is greater than or equal to a second preset threshold. For example, when m selects 1, the data amount of the data obtained by scanning according to the amplified segment time interval is 1500, and is smaller than the second preset threshold 2000, and when m selects 2, the data amount of the data obtained by scanning according to the amplified segment time interval is 2500, and is larger than the second preset threshold 2000, it may be determined that m takes a value of 2, and the segment time interval is amplified to 20.
S104: when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download.
In the embodiment of the present invention, when detecting that scanning is finished, the data export apparatus may transmit the target file to the first storage apparatus in the cloud through the sub-thread task, generate a download link of the target file through the sub-thread task, and write the download link into the second storage apparatus in the local for a user to download. In some embodiments, the first Storage device may include, but is not limited to, an Object Storage Service (OBS).
In one embodiment, when the main thread task finishes performing the scan, the data export device may notify the sub-thread task of the end of the scan by specifying a key. In some embodiments, the specified keyword may be a volaile keyword.
In the embodiment of the present invention, a data export device may obtain a data export request, where the data export request carries a screening condition, and the screening condition includes an export start time, an export end time, and a segment time interval; creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download. Scanning data from the designated database through screening condition fragments, storing the scanning result data into a preset queue, and writing the scanning result data acquired from the preset queue through the sub-thread task into a target file so as to realize simultaneous reading and writing, reduce the consumption of memory resources and improve the efficiency of data derivation.
The embodiment of the present invention also provides a data export device, which is used for executing the unit of the method in any one of the preceding claims. Specifically, referring to fig. 3, fig. 3 is a schematic block diagram of a data exporting apparatus according to an embodiment of the present invention. The data deriving device of the embodiment includes: an acquisition unit 301, a scanning unit 302, a writing unit 303, and a generation unit 304.
An obtaining unit 301, configured to obtain a data export request, where the data export request carries a screening condition, and the screening condition includes export start time, export end time, and a segment time interval;
a scanning unit 302, configured to create a main thread task according to the screening condition, scan data in an assigned database through the main thread task to obtain scanning result data, and transmit the scanning result data to a preset queue;
a writing unit 303, configured to create a sub-thread task when it is detected that there is scanning result data in the preset queue, acquire the scanning result data from the preset queue through the sub-thread task, and write the scanning result data into a target file;
and the generating unit 304 is configured to, when it is detected that the scanning is finished, transmit the target file to a first storage device in the cloud through the sub-thread task, generate a download link of the target file through the sub-thread task, and write the download link into a local second storage device for a user to download.
Further, the scanning unit 302 creates a main thread task according to the screening condition, and when the main thread task scans data in the specified database to obtain scan result data, the scanning unit is specifically configured to:
dividing the time between the export start time and the export end time into equal segments according to the segment time interval carried in the screening condition;
creating a main thread task according to each time period after the equal division and fragmentation between the export start time and the export end time;
and scanning the data in the specified database in each time period through the main thread task to obtain the scanning result data.
Further, when the scanning unit 302 scans the data in the specified database at each time period through the main thread task to obtain the scanning result data, the scanning unit is specifically configured to:
when no data is scanned in the designated database in the first time period after the equal division and fragmentation, merging the second time period and the third time period after the equal division and fragmentation;
scanning the data of the designated database in the time period after the merging processing of the second time period and the third time period;
and when data are scanned in the time periods after the merging, combining every two adjacent time periods after the second time period and the third time period, and scanning the data of the specified database according to the time periods after the combining to obtain the scanned data.
Further, the scanning unit 302 is further configured to:
when no data is scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period after the equally dividing and slicing;
scanning the data of the designated database in 4 adjacent time periods after the third time period;
and when data are scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period, and scanning the data of the specified database according to the time periods after the merging processing to obtain the scanned data.
Further, after the scanning unit 302 scans data in the specified database by the main thread task to obtain scan result data, the scanning unit is further configured to:
acquiring the data volume of the scanning result data scanned from the specified database in each time period after the equal division and fragmentation;
and adding marks to the scanning result data in each time period according to the data volume of the scanning result data in each time period, wherein the marks of the scanning data in different time periods are different.
Further, the writing unit 303 is further configured to:
when the scanning result data exist in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value;
when the fact that the data quantity of the scanning result data stored in the preset queue is larger than a first preset threshold value is monitored, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to suspend scanning of the data in the specified database;
and when the data volume of the scanning result data in the preset queue is smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of the segment of the equal segmentation, and scanning the data in the designated database according to the increased time interval of the segment of the equal segmentation.
Further, the scanning unit 302 is further configured to:
acquiring scanning progress information of scanning the data in the specified database through the main thread task;
and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In the embodiment of the present invention, a data export device may obtain a data export request, where the data export request carries a screening condition, and the screening condition includes an export start time, an export end time, and a segment time interval; creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download. Scanning data from the designated database through screening condition fragments, storing the scanning result data into a preset queue, and writing the scanning result data acquired from the preset queue through the sub-thread task into a target file so as to realize simultaneous reading and writing, reduce the consumption of memory resources and improve the efficiency of data derivation.
Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device provided in an embodiment of the present invention, and in some embodiments, the computer device in the embodiment shown in fig. 4 may include: one or more processors 401; one or more input devices 402, one or more output devices 403, and memory 404. The processor 401, the input device 402, the output device 403, and the memory 404 are connected by a bus 405. The memory 404 is used for storing computer programs and the processor 401 is used for executing computer programs stored by the memory 404. Wherein the processor 401 is configured to invoke the program to perform:
acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval;
creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download.
Further, the processor 401 creates a main thread task according to the screening condition, scans data in the specified database through the main thread task, and when obtaining scan result data, is specifically configured to:
dividing the time between the export start time and the export end time into equal segments according to the segment time interval carried in the screening condition;
creating a main thread task according to each time period after the equal division and fragmentation between the export start time and the export end time;
and scanning the data in the specified database in each time period through the main thread task to obtain the scanning result data.
Further, when the processor 401 scans the data in the specified database at each time period through the main thread task to obtain the scan result data, the processor is specifically configured to:
when no data is scanned in the designated database in the first time period after the equal division and fragmentation, merging the second time period and the third time period after the equal division and fragmentation;
scanning the data of the designated database in the time period after the merging processing of the second time period and the third time period;
and when data are scanned in the time periods after the merging, combining every two adjacent time periods after the second time period and the third time period, and scanning the data of the specified database according to the time periods after the combining to obtain the scanned data.
Further, the processor 401 is further configured to:
when no data is scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period after the equally dividing and slicing;
scanning the data of the designated database in 4 adjacent time periods after the third time period;
and when data are scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period, and scanning the data of the specified database according to the time periods after the merging processing to obtain the scanned data.
Further, after the processor 401 scans data in the specified database by the main thread task to obtain scan result data, it is further configured to:
acquiring the data volume of the scanning result data scanned from the specified database in each time period after the equal division and fragmentation;
and adding marks to the scanning result data in each time period according to the data volume of the scanning result data in each time period, wherein the marks of the scanning data in different time periods are different.
Further, the processor 401 is further configured to:
when the scanning result data exist in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value;
when the fact that the data quantity of the scanning result data stored in the preset queue is larger than a first preset threshold value is monitored, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to suspend scanning of the data in the specified database;
and when the data volume of the scanning result data in the preset queue is smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of the segment of the equal segmentation, and scanning the data in the designated database according to the increased time interval of the segment of the equal segmentation.
Further, the processor 401 is further configured to:
acquiring scanning progress information of scanning the data in the specified database through the main thread task;
and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In the embodiment of the invention, computer equipment can obtain a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval; creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download. Scanning data from the designated database through screening condition fragments, storing the scanning result data into a preset queue, and writing the scanning result data acquired from the preset queue through the sub-thread task into a target file so as to realize simultaneous reading and writing, reduce the consumption of memory resources and improve the efficiency of data derivation.
It should be understood that, in the embodiment of the present invention, the Processor 401 may be a Central Processing Unit (CPU), and the Processor may also be other general processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 404 may include a read-only memory and a random access memory, and provides instructions and data to the processor 401. A portion of the memory 404 may also include non-volatile random access memory. For example, the memory 404 may also store device type information.
In a specific implementation, the processor 401, the input device 402, and the output device 403 described in this embodiment of the present invention may execute the implementation described in the method embodiment shown in fig. 1 provided in this embodiment of the present invention, and may also execute the implementation of the data export apparatus described in fig. 3 in this embodiment of the present invention, which is not described herein again.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the data exporting method described in the embodiment corresponding to fig. 1 is implemented, and the data exporting apparatus according to the embodiment corresponding to fig. 3 of the present invention may also be implemented, which is not described herein again.
The computer readable storage medium may be an internal storage unit of the data deriving device according to any of the foregoing embodiments, for example, a hard disk or a memory of the data deriving device. The computer readable storage medium may also be an external storage device of the data exporting apparatus, such as a plug-in hard disk provided on the data exporting apparatus, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the data deriving device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the data exporting apparatus. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a computer-readable storage medium, which includes several instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes. The computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
It is emphasized that the data may also be stored in a node of a blockchain in order to further ensure the privacy and security of the data. The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
The above description is only a part of the embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.
Claims (10)
1. A method of data derivation, comprising:
acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and segment time interval;
creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
when scanning result data exist in the preset queue, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
when the scanning is detected to be finished, the target file is transmitted to a first storage device at the cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for a user to download.
2. The method according to claim 1, wherein the creating a main thread task according to the filtering condition, and scanning data in a specified database by the main thread task to obtain scanning result data comprises:
dividing the time between the export start time and the export end time into equal segments according to the segment time interval carried in the screening condition;
creating a main thread task according to each time period after the equal division and fragmentation between the export start time and the export end time;
and scanning the data in the specified database in each time period through the main thread task to obtain the scanning result data.
3. The method according to claim 2, wherein the scanning data in the specified database by the main thread task at the respective time periods to obtain the scanning result data comprises:
when no data is scanned in the designated database in the first time period after the equal division and fragmentation, merging the second time period and the third time period after the equal division and fragmentation;
scanning the data of the designated database in the time period after the merging processing of the second time period and the third time period;
and when data are scanned in the time periods after the merging, combining every two adjacent time periods after the second time period and the third time period, and scanning the data of the specified database according to the time periods after the combining to obtain the scanned data.
4. The method of claim 3, further comprising:
when no data is scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period after the equally dividing and slicing;
scanning the data of the designated database in 4 adjacent time periods after the third time period;
and when data are scanned in the time period after the merging processing, merging the 4 adjacent time periods after the third time period, and scanning the data of the specified database according to the time periods after the merging processing to obtain the scanned data.
5. The method of claim 4, wherein after the scanning data in the specified database by the main thread task to obtain the scan result data, further comprising:
acquiring the data volume of the scanning result data scanned from the specified database in each time period after the equal division and fragmentation;
and adding marks to the scanning result data in each time period according to the data volume of the scanning result data in each time period, wherein the marks of the scanning data in different time periods are different.
6. The method of claim 1, further comprising:
when the scanning result data exist in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value;
when the fact that the data quantity of the scanning result data stored in the preset queue is larger than a first preset threshold value is monitored, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to suspend scanning of the data in the specified database;
and when the data volume of the scanning result data in the preset queue is smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of the segment of the equal segmentation, and scanning the data in the designated database according to the increased time interval of the segment of the equal segmentation.
7. The method of claim 1, further comprising:
acquiring scanning progress information of scanning the data in the specified database through the main thread task;
and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
8. A data derivation apparatus, comprising:
the device comprises an acquisition unit, a data export unit and a data analysis unit, wherein the acquisition unit is used for acquiring a data export request, and the data export request carries screening conditions which comprise export start time, export end time and segment time interval;
the scanning unit is used for creating a main thread task according to the screening condition, scanning data in an appointed database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
the writing unit is used for creating a sub-thread task when the scanning result data exist in the preset queue, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
and the generating unit is used for transmitting the target file to a first storage device at the cloud end through the sub-thread task when the scanning is detected to be finished, generating a downloading link of the target file through the sub-thread task, and writing the downloading link into a local second storage device for a user to download.
9. A computer device comprising a processor and a memory, wherein the memory is configured to store a computer program and the processor is configured to invoke the computer program to perform the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110911398.4A CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110911398.4A CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113609226A true CN113609226A (en) | 2021-11-05 |
| CN113609226B CN113609226B (en) | 2024-05-14 |
Family
ID=78307834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110911398.4A Active CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113609226B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114398441A (en) * | 2022-01-21 | 2022-04-26 | 平安科技（深圳）有限公司 | Data export method, data export device, computer equipment and storage medium |
| CN114722012A (en) * | 2021-12-23 | 2022-07-08 | 武汉联影生命科学仪器有限公司 | Animal scanning imaging data exporting method and system and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN104239417A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Dynamic adjustment method and dynamic adjustment device after data fragmentation in distributed database |
| CN107256180A (en) * | 2017-05-19 | 2017-10-17 | 腾讯科技（深圳）有限公司 | Data processing method, device and terminal |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN111026768A (en) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | Data synchronization method and device capable of realizing rapid loading of data |
| CN111143462A (en) * | 2019-12-31 | 2020-05-12 | 广州酷旅旅行社有限公司 | Data export method and device, computer equipment and storage medium |
| CN112395337A (en) * | 2019-08-12 | 2021-02-23 | 北京京东振世信息技术有限公司 | Data export method and device |
| CN112445596A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Multithreading-based data import method and system and storage medium |
| CN112685365A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Data report exporting method, device, equipment and storage medium |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
-
2021
- 2021-08-09 CN CN202110911398.4A patent/CN113609226B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN104239417A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Dynamic adjustment method and dynamic adjustment device after data fragmentation in distributed database |
| CN107256180A (en) * | 2017-05-19 | 2017-10-17 | 腾讯科技（深圳）有限公司 | Data processing method, device and terminal |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN112395337A (en) * | 2019-08-12 | 2021-02-23 | 北京京东振世信息技术有限公司 | Data export method and device |
| CN111026768A (en) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | Data synchronization method and device capable of realizing rapid loading of data |
| CN111143462A (en) * | 2019-12-31 | 2020-05-12 | 广州酷旅旅行社有限公司 | Data export method and device, computer equipment and storage medium |
| CN112445596A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Multithreading-based data import method and system and storage medium |
| CN112685365A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Data report exporting method, device, equipment and storage medium |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114722012A (en) * | 2021-12-23 | 2022-07-08 | 武汉联影生命科学仪器有限公司 | Animal scanning imaging data exporting method and system and electronic equipment |
| CN114398441A (en) * | 2022-01-21 | 2022-04-26 | 平安科技（深圳）有限公司 | Data export method, data export device, computer equipment and storage medium |
| CN114398441B (en) * | 2022-01-21 | 2024-03-19 | 平安科技（深圳）有限公司 | Data export method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113609226B (en) | 2024-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113609226A (en) | Data export method and device, computer equipment and storage medium | |
| CN109325034B (en) | Data processing method, device, computer equipment and storage medium | |
| CN112527816A (en) | Data blood relationship analysis method, system, computer device and storage medium | |
| CN111061798B (en) | Configurable data transmission and monitoring method, equipment and medium | |
| CN111967023A (en) | Data encryption and decryption method, device, system and readable storage medium | |
| CN112084486A (en) | User information verification method and device, electronic equipment and storage medium | |
| CN112163412A (en) | Data verification method and device, electronic equipment and storage medium | |
| CN112434238A (en) | Webpage quality detection method and device, electronic equipment and storage medium | |
| CN115473722A (en) | Data encryption method and device, electronic equipment and storage medium | |
| CN105912664B (en) | File processing method and equipment | |
| CN113505392B (en) | Secret sharing method and device | |
| US10762207B2 (en) | Method and device for scanning virus | |
| CN114356581A (en) | Memory cleaning method and related equipment | |
| CN108520401B (en) | User list management method, device, platform and storage medium | |
| CN113132484A (en) | Data transmission method and device | |
| CN104753741A (en) | Network card test performance adjustment method and device | |
| CN109067864B (en) | Notification message pushing method and device and electronic equipment | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| CN113590298B (en) | Resource scheduling method, device, server and storage medium | |
| CN115576776A (en) | Method, system and computer readable storage medium for realizing log monitoring | |
| CN115208570A (en) | Encryption method and device based on dynamic replacement of secret key | |
| CN109901931B (en) | Reduction function quantity determination method, device and system | |
| CN110209512B (en) | Data checking method and device based on multiple data sources | |
| CN113645025A (en) | Data encryption storage method, storage device, user equipment and storage medium | |
| CN113535338A (en) | Interaction method, system, storage medium and electronic device for data access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220922 Address after: Room 2601 (Unit 07), Qianhai Free Trade Building, No. 3048, Xinghai Avenue, Nanshan Street, Qianhai Shenzhen-Hong Kong Cooperation Zone, Shenzhen, Guangdong 518000 Applicant after: Shenzhen Ping An Smart Healthcare Technology Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |