CN113609226B - Data export method and device, computer equipment and storage medium - Google Patents
Data export method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113609226B CN113609226B CN202110911398.4A CN202110911398A CN113609226B CN 113609226 B CN113609226 B CN 113609226B CN 202110911398 A CN202110911398 A CN 202110911398A CN 113609226 B CN113609226 B CN 113609226B
- Authority
- CN
- China
- Prior art keywords
- data
- scanning
- thread task
- time
- result data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012216 screening Methods 0.000 claims abstract description 52
- 239000012634 fragment Substances 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000001174 ascending effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention relates to the field of big data and discloses a data export method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a data export request, wherein the data export request carries screening conditions; creating a main thread task according to the screening conditions, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue are detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to the first storage device of the cloud end through the sub-thread task, a downloading link of the target file is generated, and the downloading link is written into the local second storage device, so that the data export efficiency is improved. The present invention relates to blockchain techniques, such as writing data into blockchains for use in data forensics and other scenarios.
Description
Technical Field
The present invention relates to the field of big data, and in particular, to a data deriving method, apparatus, computer device, and storage medium.
Background
With the increasing development of big data, there is often a scenario of exporting data in a service system, and at present, a data exporting tool is generally adopted to conduct data export, but the following defects easily exist in the implementation of data export through the data exporting tool: reading while writing cannot be realized, the data export speed is low, and when the data volume is high, the exported data occupies high memory, and even memory overflow can occur. Therefore, a solution for data export is needed to solve the above-mentioned drawbacks of data export.
Disclosure of Invention
The embodiment of the invention provides a data export method, a data export device, computer equipment and a storage medium, which can realize reading and writing by processing data in a slicing way when exporting data, reduce the consumption of memory resources, separate reading from statistics and have higher data processing speed, thereby improving the data export efficiency.
In a first aspect, an embodiment of the present invention provides a data export method, including:
acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragment time interval;
Creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
When the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
When the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user.
Further, the creating a main thread task according to the filtering condition, scanning data in a specified database through the main thread task to obtain scanning result data, including:
Dividing the time between the export start time and the export end time into equal parts according to the time intervals of the fragments carried in the screening conditions;
Creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time;
and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
Further, the scanning the data in the designated database by the main thread task in the respective time periods to obtain the scanning result data includes:
when no data is scanned in the appointed database in a first time period after dividing and slicing, combining a second time period and a third time period after dividing and slicing;
scanning data of the specified database in a time period after the second time period and the third time period are combined;
And when the data are scanned in the time period after the merging process, merging the second time period and the adjacent time period after the third time period, and scanning the data of the appointed database according to each time period after the merging process to obtain the scanning data.
Further, the method further comprises:
When the data is not scanned in the time period after the merging process, 4 adjacent time periods after the third time period after the equally dividing and slicing are merged;
scanning the data of the appointed database in a time period after combining the 4 adjacent time periods after the third time period;
And when the data is scanned in the time period after the merging process, merging the 4 adjacent time periods after the third time period, and scanning the data of the appointed database according to the time periods after the merging process to obtain the scanning data.
Further, after the data in the designated database is scanned by the main thread task to obtain the scanning result data, the method further includes:
Acquiring the data quantity of the scanning result data obtained from the specified database in a scanning way in each time period after the equal segmentation;
and adding identifiers to the scanning result data in each time period according to the data quantity of the scanning result data in each time period, wherein the identifiers of the scanning data in different time periods are different.
Further, the method further comprises:
When detecting that the scanning result data exists in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value or not;
When the data volume of the scanning result data stored in the preset queue is monitored to be larger than a first preset threshold value, a first notification message is sent to the main thread task, wherein the first notification message is used for notifying the main thread task to pause scanning of the data in the appointed database;
And when the data volume of the scanning result data in the preset queue is monitored to be smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of dividing the fragments equally, and scanning the data in the appointed database according to the increased time interval of dividing the fragments.
Further, the method further comprises:
acquiring scanning progress information of scanning data in the appointed database through the main thread task;
And storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In a second aspect, an embodiment of the present invention provides a data export apparatus, including:
An obtaining unit, configured to obtain a data export request, where the data export request carries a screening condition, and the screening condition includes an export start time, an export end time, and a segment time interval;
The scanning unit is used for creating a main thread task according to the screening condition, scanning data in a specified database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
the writing unit is used for creating a sub-thread task when the scanning result data exists in the preset queue, acquiring the scanning result data from the preset queue through the sub-thread task and writing the scanning result data into a target file;
And the generating unit is used for transmitting the target file to a first storage device of a cloud end through the sub-thread task when the scanning is finished, generating a download link of the target file through the sub-thread task, and writing the download link into a local second storage device for downloading by a user.
In a third aspect, an embodiment of the present invention provides a computer device, including a processor and a memory, where the memory is configured to store a computer program, and the processor is configured to invoke the computer program to perform the method of the first aspect.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program for execution by a processor to implement the method of the first aspect.
The embodiment of the invention can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragment time interval; creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user. By the method, when data is exported, data can be read and written by slicing, so that the consumption of memory resources is reduced, reading and statistics are separated, the data processing speed is higher, and the data exporting efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data export method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a time distribution provided by an embodiment of the present invention;
FIG. 3 is a schematic block diagram of a data deriving apparatus according to an embodiment of the present invention;
fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The data export method provided by the embodiment of the invention can be applied to a data export device, and in some embodiments, the data export device is arranged in a computer device. In certain embodiments, the computer device includes, but is not limited to, one or more of a smart phone, tablet, laptop, etc.
The embodiment of the invention can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragment time interval; creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user.
According to the embodiment of the invention, the data is scanned from the appointed database through screening the condition fragments, meanwhile, the scanning result data is stored in the preset queue, and the scanning result data obtained from the preset queue through the sub-thread task is written into the target file, so that reading while writing is realized, the consumption of memory resources is reduced, and the data deriving efficiency is improved.
The data deriving method according to the embodiment of the present invention is schematically described below with reference to fig. 1.
Referring to fig. 1, fig. 1 is a schematic flowchart of a data export method according to an embodiment of the present invention, and as shown in fig. 1, the method may be performed by a data export device, where the data export device is disposed in a computer device. Specifically, the method of the embodiment of the invention comprises the following steps.
S101: a data export request is obtained, the data export request carrying a screening condition comprising an export start time, an export end time, and a fragmented time interval.
In the embodiment of the invention, the data export device can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragmented time interval.
In some embodiments, the export start time and export end time in the screening conditions are times prior to a current time; the time interval of the segmentation is the time interval of time slicing between the export start time and the export end time, and is used for equally dividing the time between the export start time and the export end time. In some embodiments, the deriving start time, the deriving end time and the time intervals of the segments in the filtering condition are shown in fig. 2, and fig. 2 is a schematic diagram of a time distribution provided by an embodiment of the present invention.
In some embodiments, the filtering condition is used to indicate a time when data needs to be scanned from the specified database and each time period after the time is equally fragmented to create a main thread task, so that the main thread task can scan the data in the specified database in each time period. In this way, it is helpful to control the data amount and scanning speed of the data in the scan specification database by controlling the size of each period.
S102: and creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue.
In the embodiment of the invention, the data export device can create the main thread task according to the screening condition, scan the data in the appointed database through the main thread task to obtain the scanning result data, and transmit the scanning result data to the preset queue. In certain embodiments, the specified database includes, but is not limited to, a MySQL database.
In one embodiment, the data exporting device may perform equal division on the time between the export start time and the export end time according to a time interval of the segments carried in the filtering condition when creating a main thread task according to the filtering condition and scanning data in a specified database through the main thread task to obtain scanning result data; creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time; and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
In one embodiment, when the data deriving device scans the data in the specified database in each time period through the main thread task to obtain the scan result data, when no data is scanned in the specified database in a first time period after dividing equally into slices, the second time period and the third time period after dividing equally into slices may be subjected to merging processing. In one embodiment, the rule of merging the time periods after slicing is to select the number of time periods according to 2 n for merging, where n has a value of 0, 1, 2, 3.
In one embodiment, when the data in the designated database is scanned in each time period through the main thread task to obtain the scanning result data, when no data is scanned in the designated database in a first time period (namely 2 0) after dividing equally, the second time period and a third time period (2 1) after dividing equally are combined; scanning data of the specified database in a time period after the second time period and the third time period are combined; and when the data are scanned in the time period after the merging process, merging the second time period and the adjacent time period after the third time period, and scanning the data of the appointed database according to each time period after the merging process to obtain the scanning data.
In one embodiment, the data deriving means performs the merging process on 4 (2 2) adjacent time periods after the third time period after the equally-divided pieces when no data is scanned in the time period after the merging process; scanning the data of the appointed database in a time period after combining the 4 adjacent time periods after the third time period; and when the data is scanned in the time period after the merging process, merging the 4 adjacent time periods after the third time period, and scanning the data of the appointed database according to the time periods after the merging process to obtain the scanning data.
In one embodiment, when no data is scanned in the period after the merging process of the 4 adjacent periods after the third period, the merging process may be performed on 8 (i.e., 2 3) adjacent periods after the seventh period, and the data of the designated database may be scanned according to each period after the merging process.
In one embodiment, the data deriving device scans the data in the designated database through the main thread task to obtain the scan result data, and then obtains the data quantity of the scan result data scanned from the designated database in each time period after the equal segmentation; and adding identifiers to the scanning result data in each time period according to the data quantity of the scanning result data in each time period, wherein the identifiers of the scanning data in different time periods are different. In some embodiments, the data amount of the scan result data may include, but is not limited to, the number of pages, the number of bars, etc. of the scan result data.
In certain embodiments, the identification may include, but is not limited to, a numeric identification, an alphabetical identification, and the like. In one embodiment, when identifiers are added to the scan result data in each period of time, the identifiers may be added in ascending order.
In one example, if the current page is 500 pieces and the number of pieces of scan result data in each piece of time period is 500 pieces, the identifiers are added to each piece of scan result data in ascending order of numbers, and assuming that the 500 th identifier is maxId when the 500 pieces of scan result data of the current page are searched for in the scan, the maximum maxId can be transferred to the next page scan, and when the data of the next page of the current page is scanned in the piece of time period, the identifiers of the scan result data of the next page are added in ascending order of identifiers greater than maxId based on maxId.
In one embodiment, the data deriving means may obtain scan progress information for scanning data in the specified database by the main thread task; and storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device. In some embodiments, the second storage device may include, but is not limited to, a redis cache. Monitoring the scan progress is facilitated by storing the scan progress information in a second storage device.
In one embodiment, when the data deriving device transmits the scan result data to the preset queue, the data deriving device may sequentially transmit the scan result data to the preset queue according to the sequence of acquiring the scan result data.
S103: when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file.
In the embodiment of the invention, the data export device can create a sub-thread task when detecting that the scanning result data exists in the preset queue, acquire the scanning result data from the preset queue through the sub-thread task, and write the scanning result data into the target file.
In one embodiment, when the data deriving device obtains the scan result data from the preset queue through the sub-thread task, the scan result data may be obtained in batch from the preset queue through the sub-thread task.
In one embodiment, when the data deriving device writes the scan result data into the target file, the target file may be created locally by the sub-thread task, statistics is performed on the scan result data obtained from the preset queue, and the counted scan result data is written into the target file.
In one embodiment, when the sub-thread task does not acquire the scan result data from the preset queue, the sub-thread task waits until the sub-thread task acquires the scan result data from the preset queue, and the sub-thread task starts to acquire the scan result data from the preset queue and writes the scan result data into the target file.
In one embodiment, the data deriving means may monitor whether the data amount of the scan result data in the preset queue is greater than a first preset threshold value when it is detected that the scan result data exists in the preset queue; and when the data volume of the scanning result data stored in the preset queue is monitored to be larger than a first preset threshold value, sending a first notification message to the main thread task, wherein the first notification message is used for notifying the main thread task to pause scanning of the data in the appointed database. In some embodiments, the first preset threshold may be any value preset, such as 5000. In this way, it is possible to avoid excessive data in the queue due to the fact that the data reading speed is much greater than the writing speed.
In one embodiment, when the data amount of the scanning result data in the preset queue is monitored to be smaller than a second preset threshold, a second notification message is sent to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of the fragments of the equal fragments, and the data in the appointed database is scanned according to the increased time interval of the fragments. In some embodiments, the second preset threshold may be any value preset, such as 2000. In this way it is avoided that the read speed is much lower than the write speed, contributing to an increased speed of data reading.
In one embodiment, when the segment time interval of the equal segment is increased by the main thread task, the amplifying may be performed according to a preset amplifying rule according to the original segment time interval.
In one example, assuming that the original segment time interval is 5 minutes, the segment time interval may be amplified to 2 m x 5, where m has a value of 0, 1,2, 3. For example, when m selects 1, the data amount of the data scanned according to the amplified segment time interval is 1500 and is smaller than the second preset threshold 2000, when m selects 2, the data amount of the data scanned according to the amplified segment time interval is 2500 and is larger than the second preset threshold 2000, the value of m can be determined to be 2, and the segment time interval is amplified to 20.
S104: when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user.
In the embodiment of the invention, when the data export device detects that the scanning is finished, the target file is transmitted to the first storage device of the cloud end through the sub-thread task, the download link of the target file is generated through the sub-thread task, and the download link is written into the local second storage device for downloading by a user. In certain embodiments, the first storage device may include, but is not limited to, an object storage service device (Object Storage Service, OBS).
In one embodiment, when the main thread task execution scan ends, the data exporting apparatus may notify the sub thread task scan end by specifying a key. In some embodiments, the specified key may be a volatile key.
In the embodiment of the invention, the data export device can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragmented time interval; creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user. The data are scanned from the appointed database through screening condition slicing, meanwhile, the scanning result data are stored into the preset queue, and the scanning result data obtained from the preset queue through the sub-thread task are written into the target file, so that reading while writing is realized, the consumption of memory resources is reduced, and the data deriving efficiency is improved.
The embodiment of the invention also provides a data export device which is used for executing the unit of the method. Specifically, referring to fig. 3, fig. 3 is a schematic block diagram of a data deriving device according to an embodiment of the present invention. The data deriving device of the present embodiment includes: an acquisition unit 301, a scanning unit 302, a writing unit 303, and a generation unit 304.
An obtaining unit 301, configured to obtain a data export request, where the data export request carries a screening condition, and the screening condition includes an export start time, an export end time, and a fractional time interval;
The scanning unit 302 is configured to create a main thread task according to the screening condition, scan data in a specified database through the main thread task, obtain scan result data, and transmit the scan result data to a preset queue;
a writing unit 303, configured to create a sub-thread task when it is detected that there is scanning result data in the preset queue, obtain the scanning result data from the preset queue through the sub-thread task, and write the scanning result data into a target file;
And the generating unit 304 is configured to transmit, when the scanning is detected to be finished, the target file to a first storage device in the cloud end through the sub-thread task, generate, through the sub-thread task, a download link of the target file, and write the download link into a local second storage device for downloading by a user.
Further, the scanning unit 302 creates a main thread task according to the filtering condition, and scans data in a specified database through the main thread task to obtain scan result data, which is specifically configured to:
Dividing the time between the export start time and the export end time into equal parts according to the time intervals of the fragments carried in the screening conditions;
Creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time;
and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
Further, when the scanning unit 302 scans the data in the specified database in the respective time periods through the main thread task to obtain the scan result data, the scanning unit is specifically configured to:
when no data is scanned in the appointed database in a first time period after dividing and slicing, combining a second time period and a third time period after dividing and slicing;
scanning data of the specified database in a time period after the second time period and the third time period are combined;
And when the data are scanned in the time period after the merging process, merging the second time period and the adjacent time period after the third time period, and scanning the data of the appointed database according to each time period after the merging process to obtain the scanning data.
Further, the scanning unit 302 is further configured to:
When the data is not scanned in the time period after the merging process, 4 adjacent time periods after the third time period after the equally dividing and slicing are merged;
scanning the data of the appointed database in a time period after combining the 4 adjacent time periods after the third time period;
And when the data is scanned in the time period after the merging process, merging the 4 adjacent time periods after the third time period, and scanning the data of the appointed database according to the time periods after the merging process to obtain the scanning data.
Further, the scanning unit 302 is further configured to, after obtaining the scan result data by scanning the data in the designated database by the main thread task:
Acquiring the data quantity of the scanning result data obtained from the specified database in a scanning way in each time period after the equal segmentation;
and adding identifiers to the scanning result data in each time period according to the data quantity of the scanning result data in each time period, wherein the identifiers of the scanning data in different time periods are different.
Further, the writing unit 303 is further configured to:
When detecting that the scanning result data exists in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value or not;
When the data volume of the scanning result data stored in the preset queue is monitored to be larger than a first preset threshold value, a first notification message is sent to the main thread task, wherein the first notification message is used for notifying the main thread task to pause scanning of the data in the appointed database;
And when the data volume of the scanning result data in the preset queue is monitored to be smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of dividing the fragments equally, and scanning the data in the appointed database according to the increased time interval of dividing the fragments.
Further, the scanning unit 302 is further configured to:
acquiring scanning progress information of scanning data in the appointed database through the main thread task;
And storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In the embodiment of the invention, the data export device can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragmented time interval; creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user. The data are scanned from the appointed database through screening condition slicing, meanwhile, the scanning result data are stored into the preset queue, and the scanning result data obtained from the preset queue through the sub-thread task are written into the target file, so that reading while writing is realized, the consumption of memory resources is reduced, and the data deriving efficiency is improved.
Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present invention, and in some embodiments, the computer device according to the present embodiment shown in fig. 4 may include: one or more processors 401; one or more input devices 402, one or more output devices 403, and a memory 404. The processor 401, the input device 402, the output device 403, and the memory 404 are connected by a bus 405. The memory 404 is used for storing a computer program and the processor 401 is used for executing the computer program stored in the memory 404. Wherein the processor 401 is configured to invoke said program execution:
acquiring a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragment time interval;
Creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
When the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
When the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user.
Further, the processor 401 creates a main thread task according to the filtering condition, scans data in a specified database through the main thread task, and is specifically configured to:
Dividing the time between the export start time and the export end time into equal parts according to the time intervals of the fragments carried in the screening conditions;
Creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time;
and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
Further, when the processor 401 scans the data in the specified database in the respective time periods through the main thread task to obtain the scan result data, the processor is specifically configured to:
when no data is scanned in the appointed database in a first time period after dividing and slicing, combining a second time period and a third time period after dividing and slicing;
scanning data of the specified database in a time period after the second time period and the third time period are combined;
And when the data are scanned in the time period after the merging process, merging the second time period and the adjacent time period after the third time period, and scanning the data of the appointed database according to each time period after the merging process to obtain the scanning data.
Further, the processor 401 is further configured to:
When the data is not scanned in the time period after the merging process, 4 adjacent time periods after the third time period after the equally dividing and slicing are merged;
scanning the data of the appointed database in a time period after combining the 4 adjacent time periods after the third time period;
And when the data is scanned in the time period after the merging process, merging the 4 adjacent time periods after the third time period, and scanning the data of the appointed database according to the time periods after the merging process to obtain the scanning data.
Further, after the processor 401 scans the data in the designated database through the main thread task, the scan result data is further used for:
Acquiring the data quantity of the scanning result data obtained from the specified database in a scanning way in each time period after the equal segmentation;
and adding identifiers to the scanning result data in each time period according to the data quantity of the scanning result data in each time period, wherein the identifiers of the scanning data in different time periods are different.
Further, the processor 401 is further configured to:
When detecting that the scanning result data exists in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value or not;
When the data volume of the scanning result data stored in the preset queue is monitored to be larger than a first preset threshold value, a first notification message is sent to the main thread task, wherein the first notification message is used for notifying the main thread task to pause scanning of the data in the appointed database;
And when the data volume of the scanning result data in the preset queue is monitored to be smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of dividing the fragments equally, and scanning the data in the appointed database according to the increased time interval of dividing the fragments.
Further, the processor 401 is further configured to:
acquiring scanning progress information of scanning data in the appointed database through the main thread task;
And storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
In the embodiment of the invention, the computer equipment can acquire a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a fragmented time interval; creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue; when the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file; when the scanning is detected to be finished, the target file is transmitted to a first storage device of a cloud end through the sub-thread task, a downloading link of the target file is generated through the sub-thread task, and the downloading link is written into a local second storage device for downloading by a user. The data are scanned from the appointed database through screening condition slicing, meanwhile, the scanning result data are stored into the preset queue, and the scanning result data obtained from the preset queue through the sub-thread task are written into the target file, so that reading while writing is realized, the consumption of memory resources is reduced, and the data deriving efficiency is improved.
It should be appreciated that in embodiments of the present invention, the Processor 401 may be a central processing unit (CenSral Processing UniS, CPU), but may also be other general purpose processors, digital signal processors (DIGISAL SIGNAL processors, DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GASE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 402 may include a touch pad, microphone, etc., and the output device 403 may include a display (LCD, etc.), speaker, etc.
The memory 404 may include read only memory and random access memory and provide instructions and data to the processor 401. A portion of memory 404 may also include non-volatile random access memory. For example, memory 404 may also store information of device type.
In a specific implementation, the processor 401, the input device 402, and the output device 403 described in the embodiments of the present invention may execute the implementation described in the embodiment of the method described in fig. 1 provided in the embodiments of the present invention, and may also execute the implementation of the data deriving apparatus described in fig. 3 in the embodiments of the present invention, which is not described herein again.
The embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program when executed by a processor implements the data export method described in the embodiment corresponding to fig. 1, and may also implement the data export device of the embodiment corresponding to fig. 3, which is not described herein again.
The computer readable storage medium may be an internal storage unit of the data deriving device according to any of the above embodiments, for example, a hard disk or a memory of the data deriving device. The computer-readable storage medium may be an external storage device of the data deriving device, such as a plug-in hard disk, a smart memory card (SMARS MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like, which are provided on the data deriving device. Further, the computer readable storage medium may further include both an internal storage unit and an external storage device of the data deriving device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the data deriving means. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a computer-readable storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a terminal, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes. The computer readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created from the use of blockchain nodes, and the like.
It is emphasized that to further guarantee the privacy and security of the data, the data may also be stored in a blockchain node. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The blockchain (Blockchain), essentially a de-centralized database, is a string of data blocks that are generated in association using cryptographic methods, each of which contains information from a batch of network transactions for verifying the validity (anti-counterfeit) of its information and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention.
Claims (9)
1. A data export method, comprising:
Obtaining a data export request, wherein the data export request carries screening conditions, and the screening conditions comprise export start time, export end time and a segmented time interval, wherein the segmented time interval is a time slicing time interval between the export start time and the export end time;
Creating a main thread task according to the screening condition, scanning data in a designated database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
When the scanning result data in the preset queue is detected, creating a sub-thread task, acquiring the scanning result data from the preset queue through the sub-thread task, and writing the scanning result data into a target file;
When the scanning is detected to be finished, transmitting the target file to a first storage device of a cloud end through the sub-thread task, generating a download link of the target file through the sub-thread task, and writing the download link into a local second storage device for downloading by a user;
Creating a main thread task according to the screening condition, scanning data in a specified database through the main thread task to obtain scanning result data, wherein the method comprises the following steps:
Dividing the time between the export start time and the export end time into equal parts according to the time intervals of the fragments carried in the screening conditions;
Creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time;
and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
2. The method of claim 1, wherein the scanning data in the specified database by the main thread task for the respective time periods to obtain the scan result data comprises:
when no data is scanned in the appointed database in a first time period after dividing and slicing, combining a second time period and a third time period after dividing and slicing;
scanning data of the specified database in a time period after the second time period and the third time period are combined;
And when the data are scanned in the time period after the merging processing, merging the second time period and the adjacent time period after the third time period, and scanning the data of the appointed database according to each time period after the merging processing to obtain the scanning result data.
3. The method according to claim 2, wherein the method further comprises:
When the data is not scanned in the time period after the merging process, 4 adjacent time periods after the third time period after the equally dividing and slicing are merged;
scanning the data of the appointed database in a time period after combining the 4 adjacent time periods after the third time period;
And when the data is scanned in the time period after the merging process, merging the 4 adjacent time periods after the third time period, and scanning the data of the appointed database according to the time periods after the merging process to obtain the scanning result data.
4. A method according to claim 3, wherein said scanning of data in a designated database by said main thread task, after obtaining scan result data, further comprises:
Acquiring the data quantity of the scanning result data obtained from the specified database in a scanning way in each time period after the equal segmentation;
And adding identifiers to the scanning result data in each time period according to the data quantity of the scanning result data in each time period, wherein the identifiers of the scanning result data in different time periods are different.
5. The method according to claim 1, wherein the method further comprises:
When detecting that the scanning result data exists in the preset queue, monitoring whether the data quantity of the scanning result data in the preset queue is larger than a first preset threshold value or not;
When the data volume of the scanning result data stored in the preset queue is monitored to be larger than a first preset threshold value, a first notification message is sent to the main thread task, wherein the first notification message is used for notifying the main thread task to pause scanning of the data in the appointed database;
And when the data volume of the scanning result data in the preset queue is monitored to be smaller than a second preset threshold value, sending a second notification message to the main thread task, wherein the second notification message is used for notifying the main thread task to increase the time interval of dividing the fragments equally, and scanning the data in the appointed database according to the increased time interval of dividing the fragments.
6. The method according to claim 1, wherein the method further comprises:
acquiring scanning progress information of scanning data in the appointed database through the main thread task;
And storing the scanning progress information into the second storage device, wherein the second storage device is a cache storage device.
7. A data deriving apparatus, comprising:
An obtaining unit, configured to obtain a data export request, where the data export request carries a screening condition, where the screening condition includes an export start time, an export end time, and a fractional time interval, where the fractional time interval is a time interval of time slicing between the export start time and the export end time;
The scanning unit is used for creating a main thread task according to the screening condition, scanning data in a specified database through the main thread task to obtain scanning result data, and transmitting the scanning result data to a preset queue;
the writing unit is used for creating a sub-thread task when the scanning result data exists in the preset queue, acquiring the scanning result data from the preset queue through the sub-thread task and writing the scanning result data into a target file;
The generating unit is used for transmitting the target file to a first storage device of a cloud end through the sub-thread task when the scanning is finished, generating a download link of the target file through the sub-thread task, and writing the download link into a local second storage device for downloading by a user;
The scanning unit creates a main thread task according to the screening conditions, scans data in a specified database through the main thread task, and is particularly used for equally dividing and slicing the time between the export starting time and the export ending time according to the segmentation time interval carried in the screening conditions when scanning result data is obtained; creating a main thread task according to each time period after dividing the slices equally between the export starting time and the export ending time; and scanning the data in the appointed database in each time period through the main thread task to obtain the scanning result data.
8. A computer device comprising a processor and a memory, wherein the memory is for storing a computer program, the processor being configured to invoke the computer program to perform the method of any of claims 1-6.
9. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which is executed by a processor to implement the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110911398.4A CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110911398.4A CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113609226A CN113609226A (en) | 2021-11-05 |
| CN113609226B true CN113609226B (en) | 2024-05-14 |
Family
ID=78307834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110911398.4A Active CN113609226B (en) | 2021-08-09 | 2021-08-09 | Data export method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113609226B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114398441B (en) * | 2022-01-21 | 2024-03-19 | 平安科技（深圳）有限公司 | Data export method, device, computer equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN104239417A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Dynamic adjustment method and dynamic adjustment device after data fragmentation in distributed database |
| CN107256180A (en) * | 2017-05-19 | 2017-10-17 | 腾讯科技（深圳）有限公司 | Data processing method, device and terminal |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN111026768A (en) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | Data synchronization method and device capable of realizing rapid loading of data |
| CN111143462A (en) * | 2019-12-31 | 2020-05-12 | 广州酷旅旅行社有限公司 | Data export method and device, computer equipment and storage medium |
| CN112395337A (en) * | 2019-08-12 | 2021-02-23 | 北京京东振世信息技术有限公司 | Data export method and device |
| CN112445596A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Multithreading-based data import method and system and storage medium |
| CN112685365A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Data report exporting method, device, equipment and storage medium |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
-
2021
- 2021-08-09 CN CN202110911398.4A patent/CN113609226B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN104239417A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Dynamic adjustment method and dynamic adjustment device after data fragmentation in distributed database |
| CN107256180A (en) * | 2017-05-19 | 2017-10-17 | 腾讯科技（深圳）有限公司 | Data processing method, device and terminal |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN112395337A (en) * | 2019-08-12 | 2021-02-23 | 北京京东振世信息技术有限公司 | Data export method and device |
| CN111026768A (en) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | Data synchronization method and device capable of realizing rapid loading of data |
| CN111143462A (en) * | 2019-12-31 | 2020-05-12 | 广州酷旅旅行社有限公司 | Data export method and device, computer equipment and storage medium |
| CN112445596A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Multithreading-based data import method and system and storage medium |
| CN112685365A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Data report exporting method, device, equipment and storage medium |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113609226A (en) | 2021-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107733968B (en) | Application information pushing method and device, computer equipment and storage medium | |
| US9081838B2 (en) | Methods, apparatuses, and computer program products for database record recovery | |
| US9355250B2 (en) | Method and system for rapidly scanning files | |
| US20200021665A1 (en) | Method and device for timeout monitoring | |
| CN112527414B (en) | Front-end-based data processing method, device, equipment and storage medium | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| CN113609226B (en) | Data export method and device, computer equipment and storage medium | |
| CN111061798A (en) | Configurable data transmission and monitoring method, equipment and medium | |
| CN112163412A (en) | Data verification method and device, electronic equipment and storage medium | |
| JP2019519849A (en) | Method and device for preventing attacks on servers | |
| CN111898036A (en) | Behavior data collecting and processing system and method | |
| US9648086B2 (en) | System, method, and computer program product for providing test logic to user devices | |
| US10762207B2 (en) | Method and device for scanning virus | |
| US20150002680A1 (en) | System, method, and computer program product for receiving and executing test logic at user devices | |
| RU2731122C1 (en) | Method, system and device for displaying text | |
| CN113780580B (en) | Data analysis method, device, equipment and storage medium based on machine learning | |
| CN111354383B (en) | Audio defect positioning method and device and terminal equipment | |
| CN113050918A (en) | Audio optimization method, device, equipment and storage medium based on remote double recording | |
| WO2020065778A1 (en) | Information processing device, control method, and program | |
| CN113590298B (en) | Resource scheduling method, device, server and storage medium | |
| CN114371866A (en) | Version reconfiguration test method, device and equipment of service system | |
| CN114629895B (en) | File fragment breakpoint continuous transmission method, device, terminal equipment and medium | |
| JP7268742B2 (en) | Policy evaluation device, control method, and program | |
| CN114221816B (en) | Flow detection method, device, equipment and storage medium | |
| CN114095203A (en) | Data encryption interaction method and device and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220922 Address after: Room 2601 (Unit 07), Qianhai Free Trade Building, No. 3048, Xinghai Avenue, Nanshan Street, Qianhai Shenzhen-Hong Kong Cooperation Zone, Shenzhen, Guangdong 518000 Applicant after: Shenzhen Ping An Smart Healthcare Technology Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |