CN117389526A - Data offline export method and device, electronic equipment and storage medium - Google Patents

Data offline export method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN117389526A
CN117389526A CN202311398257.2A CN202311398257A CN117389526A CN 117389526 A CN117389526 A CN 117389526A CN 202311398257 A CN202311398257 A CN 202311398257A CN 117389526 A CN117389526 A CN 117389526A
Authority
CN
China
Prior art keywords
data
offline
downloading
export
download
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311398257.2A
Other languages
Chinese (zh)
Inventor
杨壮
仲雪莲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Bank Co Ltd
Original Assignee
Ping An Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Bank Co Ltd filed Critical Ping An Bank Co Ltd
Priority to CN202311398257.2A priority Critical patent/CN117389526A/en
Publication of CN117389526A publication Critical patent/CN117389526A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/20—Software design
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application provides a data offline export method, a device, electronic equipment and a storage medium, wherein the data offline export method comprises the following steps: responding to a data export request triggered by a user aiming at a target database; determining the total amount of the downloaded data based on the data total calculation interface and the query condition; judging whether the total amount of the downloaded data is larger than an offline threshold value; and when the total amount of the downloaded data is greater than or equal to the offline threshold value, creating an offline download task based on the query condition and the data body, adding the offline download task into an offline download table, otherwise, creating an real-time download task based on the query condition and the data body, and starting an export queue to execute the download task based on the export queue. The method and the device can be compatible with two data export modes of real-time downloading and offline downloading. Meanwhile, the method and the device do not need to carry out secondary development on all processing procedures, and can improve development efficiency and reduce code difference.

Description

Data offline export method and device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of computers, and in particular, to a method and apparatus for offline exporting data, an electronic device, and a storage medium.
Background
At present, the existing export function is basically realized based on POI technology, wherein the realization mode based on POI technology requires each developer to additionally compile a processing layer in the execution process of the download task, so that a lot of manpower and material resources are consumed, bug is easy to output, development efficiency is low, different developers have different development habits, the code cannot be unified, and the learning cost of the developers is increased. In addition, existing below modes are not compatible with both real-time and offline modes.
Disclosure of Invention
An objective of the embodiments of the present application is to provide a method, an apparatus, an electronic device, and a storage medium for offline exporting data, so as to realize data downloading, which is compatible with two data exporting modes of real-time downloading and offline downloading. Meanwhile, when the method is used, only the header of the query condition and the data body of the instruction processing logic are required by the developer, and secondary development is not required to be carried out on all the processing processes, so that the workload is reduced, the development efficiency is improved, and in addition, the problem that the code difference is large due to different styles of different developers in the secondary development process can be avoided.
In a first aspect, the present invention provides a method for offline exporting data, the method comprising:
responding to a data export request triggered by a user aiming at a target database, wherein the data export request carries a downloading parameter, and the downloading parameter comprises a header indicating a query condition and a data body indicating processing logic;
determining the total amount of downloaded data based on a data total calculation interface and the query condition, wherein the query condition is used as an input parameter of the data total calculation interface, and the data total calculation interface returns the total amount of downloaded data based on the query condition;
judging whether the total amount of the downloaded data is larger than an offline threshold value or not;
when the total amount of the downloaded data is greater than or equal to the offline threshold value, an offline downloading task is created based on the query condition and the data body, and the offline downloading task is added into an offline downloading table;
and when the total amount of the downloaded data is smaller than the offline threshold value, creating an in-situ downloading task based on the query condition and the data body, and starting an export queue to execute the downloading task based on the export queue, wherein the export queue executes the downloading task based on a thread pool.
According to the method, through responding to the data export request triggered by a user aiming at the target database, the total download data amount can be determined based on the data total number calculation interface and the query condition, whether the total download data amount is larger than the offline threshold value can be judged, and further, when the total download data amount is larger than or equal to the offline threshold value, an offline download task can be created based on the query condition and the data body and added into an offline download table, and when the total download data amount is smaller than the offline threshold value, an actual download task can be created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue, wherein the export queue executes the download task based on a thread pool, and therefore, based on the offline threshold value, two modes of actual download and offline download are compatible. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
In an alternative embodiment, the method further comprises:
creating a timer and polling the offline download table based on the timer to judge whether the offline download table has a download task to be processed, and if so, starting the export queue to execute the download task to be processed, wherein the timer polls the offline download table every 2 minutes.
This alternative embodiment is capable of performing all pending download tasks in the offline download table based on a timer polling scheme.
In an alternative embodiment, the method further comprises:
dividing the downloading task into at least two sub-downloading tasks based on a first preset threshold value, and executing the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
The optional implementation manner can divide the downloading task into at least two sub-downloading tasks based on a first preset threshold value, so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
In an alternative embodiment, the method further comprises:
acquiring target data obtained by the export queue executing the downloading task;
and storing the target data in a plurality of files based on a second preset threshold.
The optional implementation manner can automatically obtain the target data obtained by executing the downloading task by the export queue, and then can automatically store the target data in a plurality of files based on a second preset threshold.
In a second aspect, the present invention provides an offline data exporting apparatus, the apparatus comprising:
the response module is used for responding to a data export request triggered by a user aiming at a target database, wherein the data export request carries a downloading parameter, and the downloading parameter comprises a table head indicating a query condition and a data body indicating processing logic;
the calculation module is used for determining the total amount of the downloaded data based on a data total number calculation interface and the query condition, wherein the query condition is used as an input parameter of the data total number calculation interface, and the data total number calculation interface returns the total amount of the downloaded data based on the query condition;
the judging module is used for judging whether the total amount of the downloaded data is larger than an offline threshold value or not;
the first creating module is used for creating an offline downloading task based on the query condition and the data body when the total amount of the downloading data is larger than or equal to the offline threshold value, and adding the offline downloading task into an offline downloading table;
and the second creating module is used for creating an actual downloading task based on the query condition and the data body when the total amount of the downloading data is smaller than the offline threshold value, and starting an export queue to execute the downloading task based on the export queue, wherein the export queue executes the downloading task based on a thread pool.
According to the device, through responding to the data export request triggered by a user aiming at the target database, the total download data amount can be determined based on the data total number calculation interface and the query condition, whether the total download data amount is larger than the offline threshold value can be judged, and further, when the total download data amount is larger than or equal to the offline threshold value, an offline download task can be created based on the query condition and the data body and added into an offline download table, and when the total download data amount is smaller than the offline threshold value, an actual download task can be created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue, wherein the export queue executes the download task based on a thread pool, and therefore, based on the offline threshold value, two modes of actual download and offline download are compatible. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
In an alternative embodiment, the apparatus further comprises:
and the third creating module is used for creating a timer and polling the offline download table based on the timer so as to judge whether the offline download table has a download task to be processed, and if the offline download task to be processed exists, starting the export queue to execute the download task to be processed, wherein the timer polls the offline download table every 2 minutes.
This alternative embodiment is capable of performing all pending download tasks in the offline download table based on a timer polling scheme.
In an alternative embodiment, the apparatus further comprises:
the task dividing module is used for dividing the downloading task into at least two sub-downloading tasks based on a first preset threshold value so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
The optional implementation manner can divide the downloading task into at least two sub-downloading tasks based on a first preset threshold value, so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
In an alternative embodiment, the apparatus further comprises:
the acquisition module is used for acquiring target data obtained by executing the downloading task by the export queue;
and the storage module is used for storing the target data into a plurality of files based on a second preset threshold value.
The optional implementation manner can automatically obtain the target data obtained by executing the downloading task by the export queue, and then can automatically store the target data in a plurality of files based on a second preset threshold.
In a third aspect, the present invention provides an electronic device comprising:
a processor; and
a memory configured to store machine readable instructions that, when executed by the processor, perform the data offline export method of any of the preceding embodiments.
According to the device, through executing the data offline export method, a data export request triggered by a user aiming at a target database can be responded, the total download data amount can be determined based on a data total calculation interface and the query condition, whether the total download data amount is larger than an offline threshold can be judged, and further when the total download data amount is larger than or equal to the offline threshold, an offline download task can be created based on the query condition and the data body and added into an offline download table, and when the total download data amount is smaller than the offline threshold, an actual download task can be created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
In a fourth aspect, the present invention provides a storage medium storing a computer program for executing the data offline export method according to any of the previous embodiments by a processor.
According to the storage medium, through executing the data offline export method, a data export request triggered by a user aiming at a target database can be responded, the total download data amount can be determined based on a data total calculation interface and the query condition, whether the total download data amount is larger than an offline threshold can be judged, and further, when the total download data amount is larger than or equal to the offline threshold, an offline download task can be created based on the query condition and the data body and added into an offline download table, and when the total download data amount is smaller than the offline threshold, an actual download task can be created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for offline exporting data according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a data offline exporting device according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
Example 1
Referring to fig. 1, fig. 1 is a flowchart of a data offline export method disclosed in an embodiment of the present application, and as shown in fig. 1, the method in the embodiment of the present application includes the following steps:
101. responding to a data export request triggered by a user aiming at a target database, wherein the data export request carries a downloading parameter, and the downloading parameter comprises a header indicating a query condition and a data body indicating processing logic;
102. determining the total amount of the downloaded data based on the data total number calculation interface and the query condition, wherein the query condition is used as an input parameter of the data total number calculation interface, and the data total number calculation interface returns the total amount of the downloaded data based on the query condition;
103. judging whether the total amount of the downloaded data is larger than an offline threshold value;
104. when the total amount of the downloaded data is greater than or equal to an offline threshold value, an offline downloading task is created based on the query condition and the data body, and the offline downloading task is added into an offline downloading table;
105. when the total amount of the downloaded data is smaller than the offline threshold, creating an in-situ downloading task based on the query condition and the data body, and starting an export queue to execute the downloading task based on the export queue, wherein the export queue executes the downloading task based on the thread pool.
According to the method, through responding to the data export request triggered by the user aiming at the target database, the total download data amount can be determined based on the data total number calculation interface and the query condition, whether the total download data amount is larger than the offline threshold can be judged, and further when the total download data amount is larger than or equal to the offline threshold, an offline download task is created based on the query condition and the data body and added into the offline download table, and when the total download data amount is smaller than the offline threshold, an actual download task is created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue, wherein the export queue executes the download task based on the thread pool, and therefore the method is compatible with two modes of real-time download and offline download based on the offline threshold. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
For the embodiment of the application, as an example, assuming that the user needs to derive the data about staff in the Oracle database, the user defines the header "name", "team", "age", and defines the data body for processing the "name", "team", "age", wherein the data body can splice the "name", "job position", "job age" into the following character string [ name: small; team A; age:25].
In this embodiment of the present application, for step 101, specifically, the user may trigger a data export request through the user terminal, where the download parameter carried by the data export request may include parameters such as an access path of the target database, a download speed setting value, and the like, in addition to a header indicating a query condition and a data body indicating processing logic.
In this embodiment of the present application, for step 101, specifically, the data body indicating the processing logic refers to logic that performs preprocessing on data after acquiring the data, for example, it may be character stitching, or generate a prompt message of the data acquisition state.
In the embodiment of the present application, the target database may be a Mysql, oracle, elasticsarch database for step 101, which is not limited in the embodiment of the present application.
In this embodiment of the present application, for step 102, the data total calculation interface may use the query condition as an input parameter of the data total calculation interface, so as to determine the total amount of downloaded data, for example, when the query condition indicates that "name", "team", and "age" need to be acquired, it may query the data table with the header of "name", "team", and "age" for the number of data pieces recorded in the data table, so that the number of data pieces is taken as the total amount of downloaded data, and if it is assumed that the data table is recorded with 1000 pieces of data, the total amount of downloaded data is 1000 pieces.
In this embodiment of the present application, for step 103, the offline threshold is used to determine what downloading method is created, where the offline threshold may be 1G, or may be another value, for example, when the total amount of downloaded data exceeds 1G, since it takes a long time, downloading is performed in an offline manner. Further, the offline threshold may be set by a user.
In the embodiment of the present application, for step 105, the export queue may download a process, where the download process manages a thread pool composed of multiple threads, and thus is capable of executing a download task based on the thread pool.
In an embodiment of the present application, as an optional implementation manner, the method of the embodiment of the present application further includes the following steps:
creating a timer and polling the offline download table based on the timer to judge whether the offline download table has a to-be-processed download task, and if so, starting a export queue to execute the to-be-processed download task, wherein the timer polls the offline download table every 2 minutes.
This alternative embodiment is capable of performing all pending download tasks in the offline download table based on a timer polling scheme.
For the above alternative embodiments, reference is made to the prior art for a detailed description of the timer.
In an embodiment of the present application, in an optional implementation manner, the method of the embodiment of the present application further includes the following steps:
dividing the downloading task into at least two sub-downloading tasks based on a first preset threshold value, and executing the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
The optional implementation manner can divide the downloading task into at least two sub-downloading tasks based on a first preset threshold value, so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
For the above alternative embodiment, the first preset threshold may be 10000 or 20000, where the specific value thereof may be determined based on the type of the target database, for example 10000 in the elastic search database and 20000 in the Oracle or Mysql. As an example, assuming that the target database is an Oracle database and 40000 pieces of data need to be acquired from the Oracle database, the entire download task may be divided into 2 sub-download tasks, each downloading 20000 pieces of data.
In an embodiment of the present application, as an optional implementation manner, the method of the embodiment of the present application further includes the following steps:
obtaining target data obtained by executing a downloading task by a export queue;
and storing the target data in a plurality of files based on the second preset threshold.
The optional implementation manner can automatically acquire the target data obtained by the export queue executing the downloading task, and then can automatically store the target data into a plurality of files based on the second preset threshold.
For the above alternative embodiment, the second preset threshold may be 200M, for example, assuming that a sub-download task downloads 20000 pieces of data for 600M, the 20000 pieces of data are saved in 3 files.
For the above alternative embodiment, further, the file may be saved in a file directory specified by the user, and the address of the file directory may be saved in an offline record, where the file directory may be specified by the user.
For the alternative embodiments described above, the export queue may initiate a program to build a file to create the file.
Example two
Referring to fig. 2, fig. 2 is a schematic structural diagram of an offline data exporting device disclosed in an embodiment of the present application, and as shown in fig. 2, the device in the embodiment of the present application includes the following functional modules:
a response module 201, configured to respond to a data export request triggered by a user for a target database, where the data export request carries a download parameter, and the download parameter includes a header indicating a query condition and a data body indicating processing logic;
a calculation module 202, configured to determine a total amount of downloaded data based on a total number of data calculation interfaces and a query condition, where the query condition is used as an input parameter of the total number of data calculation interfaces, and the total number of data calculation interfaces returns the total amount of downloaded data based on the query condition;
a judging module 203, configured to judge whether the total amount of the downloaded data is greater than an offline threshold;
the first creating module 204 is configured to create an offline download task based on the query condition and the data body when the total amount of the download data is greater than or equal to the offline threshold, and add the offline download task to the offline download table;
and a second creating module 205, configured to create an in-situ download task based on the query condition and the data body when the total amount of the download data is less than the offline threshold, and open an export queue to perform the download task based on the export queue, where the export queue performs the download task based on the thread pool.
According to the device, through responding to the data export request triggered by the user aiming at the target database, the total data quantity can be calculated based on the interface and the query condition, the total download data quantity can be determined, whether the total download data quantity is larger than the offline threshold value can be judged, and further when the total download data quantity is larger than or equal to the offline threshold value, an offline download task is created based on the query condition and the data body and added into the offline download table, and when the total download data quantity is smaller than the offline threshold value, an actual download task is created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue, wherein the export queue executes the download task based on the thread pool, and therefore, the device is compatible with two modes of real-time download and offline download based on the offline threshold value. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
In this embodiment of the present application, as an optional implementation manner, the apparatus of this embodiment of the present application further includes the following functional modules:
and the third creating module is used for creating a timer and polling the offline download table based on the timer so as to judge whether the offline download table has a download task to be processed, and if the offline download table has the download task to be processed, starting the export queue to execute the download task to be processed, wherein the timer polls the offline download table every 2 minutes.
This alternative embodiment is capable of performing all pending download tasks in the offline download table based on a timer polling scheme.
In this embodiment of the present application, as an optional implementation manner, the apparatus of this embodiment of the present application further includes a function module task dividing module configured to divide the download task into at least two sub-download tasks based on a first preset threshold, so as to execute the sub-download tasks based on the export queue, where a download data amount of the sub-download tasks is smaller than a download data total amount corresponding to the download task.
The optional implementation manner can divide the downloading task into at least two sub-downloading tasks based on a first preset threshold value, so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
In this embodiment of the present application, as an optional implementation manner, the apparatus of this embodiment of the present application further includes a function module obtaining module, configured to obtain target data obtained by performing a download task by using a export queue;
and the storage module is used for storing the target data into a plurality of files based on the second preset threshold value.
The optional implementation manner can automatically acquire the target data obtained by the export queue executing the downloading task, and then can automatically store the target data into a plurality of files based on the second preset threshold.
It should be noted that, for other detailed descriptions of the apparatus in the embodiments of the present application, please refer to the related descriptions in the first embodiment of the present application, which are not repeated herein.
Example III
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application, and as shown in fig. 3, the electronic device in the embodiment of the present application includes:
a processor 301; and
a memory 302 configured to store machine readable instructions that, when executed by the processor 301, perform a data offline export method as in any of the previous embodiments.
According to the electronic device, through executing the data offline export method, a data export request triggered by a user aiming at a target database can be responded, the total amount of downloaded data can be determined based on the data total number calculation interface and the query condition, whether the total amount of downloaded data is larger than an offline threshold value can be judged, and further when the total amount of downloaded data is larger than or equal to the offline threshold value, an offline download task is created based on the query condition and the data body and added into an offline download table, and when the total amount of downloaded data is smaller than the offline threshold value, an actual download task is created based on the query condition and the data body and an export queue is started to execute the download task based on the export queue. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
Example IV
The present embodiments disclose a storage medium storing a computer program that is executed by a processor to perform the data offline export method of any of the previous embodiments.
According to the storage medium, through executing the data offline export method, a data export request triggered by a user aiming at a target database can be responded, the total amount of downloaded data can be determined based on a data total number calculation interface and query conditions, whether the total amount of downloaded data is larger than an offline threshold value can be judged, and further when the total amount of downloaded data is larger than or equal to the offline threshold value, an offline download task is created based on the query conditions and the data body and added into an offline download table, and when the total amount of downloaded data is smaller than the offline threshold value, an actual download task is created based on the query conditions and the data body and an export queue is started to execute the download task based on the export queue. Meanwhile, the user only needs to customize the download parameters, and the secondary development is not needed for most of processing logic, so that the code difference is reduced, the learning cost is reduced, and the workload of developers is reduced.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
Further, the units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Furthermore, functional modules in various embodiments of the present application may be integrated together to form a single portion, or each module may exist alone, or two or more modules may be integrated to form a single portion.
It should be noted that the functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM) random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above is only an example of the present application, and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims (10)

1. A method for offline exporting data, the method comprising:
responding to a data export request triggered by a user aiming at a target database, wherein the data export request carries a downloading parameter, and the downloading parameter comprises a header indicating a query condition and a data body indicating processing logic;
determining the total amount of downloaded data based on a data total calculation interface and the query condition, wherein the query condition is used as an input parameter of the data total calculation interface, and the data total calculation interface returns the total amount of downloaded data based on the query condition;
judging whether the total amount of the downloaded data is larger than an offline threshold value or not;
when the total amount of the downloaded data is greater than or equal to the offline threshold value, an offline downloading task is created based on the query condition and the data body, and the offline downloading task is added into an offline downloading table;
and when the total amount of the downloaded data is smaller than the offline threshold, creating an in-situ downloading task based on the query condition and the data body, and starting an export queue to execute the downloading task based on the export queue, wherein the export queue executes the downloading task based on a thread pool.
2. The method of claim 1, wherein the method further comprises:
creating a timer and polling the offline download table based on the timer to judge whether the offline download table has a download task to be processed, and if so, starting the export queue to execute the download task to be processed, wherein the timer polls the offline download table every 2 minutes.
3. The method of claim 1, wherein the method further comprises:
dividing the downloading task into at least two sub-downloading tasks based on a first preset threshold value, and executing the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
4. The method of claim 1, wherein the method further comprises:
acquiring target data obtained by the export queue executing the downloading task;
and storing the target data in a plurality of files based on a second preset threshold.
5. An apparatus for off-line exporting of data, the apparatus comprising:
the response module is used for responding to a data export request triggered by a user aiming at a target database, wherein the data export request carries a downloading parameter, and the downloading parameter comprises a table head indicating a query condition and a data body indicating processing logic;
the calculation module is used for determining the total amount of the downloaded data based on a data total number calculation interface and the query condition, wherein the query condition is used as an input parameter of the data total number calculation interface, and the data total number calculation interface returns the total amount of the downloaded data based on the query condition;
the judging module is used for judging whether the total amount of the downloaded data is larger than an offline threshold value or not;
the first creating module is used for creating an offline downloading task based on the query condition and the data body when the total amount of the downloading data is larger than or equal to the offline threshold value, and adding the offline downloading task into an offline downloading table;
and the second creating module is used for creating an actual downloading task based on the query condition and the data body when the total amount of the downloading data is smaller than the offline threshold value, and starting an export queue to execute the downloading task based on the export queue, wherein the export queue executes the downloading task based on a thread pool.
6. The apparatus of claim 5, wherein the apparatus further comprises:
and the third creating module is used for creating a timer and polling the offline download table based on the timer so as to judge whether the offline download table has a download task to be processed, and if the offline download task to be processed exists, starting the export queue to execute the download task to be processed, wherein the timer polls the offline download table every 2 minutes.
7. The apparatus of claim 5, wherein the apparatus further comprises:
the task dividing module is used for dividing the downloading task into at least two sub-downloading tasks based on a first preset threshold value so as to execute the sub-downloading tasks based on the export queue, wherein the downloading data volume of the sub-downloading tasks is smaller than the downloading data volume corresponding to the downloading tasks.
8. The apparatus of claim 5, wherein the apparatus further comprises:
the acquisition module is used for acquiring target data obtained by executing the downloading task by the export queue;
and the storage module is used for storing the target data into a plurality of files based on a second preset threshold value.
9. An electronic device, comprising:
a processor; and
a memory configured to store machine readable instructions that, when executed by the processor, perform the data offline export method of any of claims 1-4.
10. A storage medium storing a computer program for execution by a processor of the data offline export method according to any of claims 1-4.
CN202311398257.2A 2023-10-25 2023-10-25 Data offline export method and device, electronic equipment and storage medium Pending CN117389526A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311398257.2A CN117389526A (en) 2023-10-25 2023-10-25 Data offline export method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311398257.2A CN117389526A (en) 2023-10-25 2023-10-25 Data offline export method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117389526A true CN117389526A (en) 2024-01-12

Family

ID=89466362

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311398257.2A Pending CN117389526A (en) 2023-10-25 2023-10-25 Data offline export method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117389526A (en)

Similar Documents

Publication Publication Date Title
US20230351145A1 (en) Pipelining and parallelizing graph execution method for neural network model computation and apparatus thereof
US8561032B2 (en) Visualizing thread life time in eclipse
CN111367692A (en) Search engine data processing method and device, electronic equipment and medium
CN114281653B (en) Application program monitoring method and device and computing equipment
CN109063040B (en) Client program data acquisition method and system
CN114448972A (en) Distributed storage log compression downloading method, system, terminal and storage medium
CN112948068B (en) Task scheduling method and device and electronic equipment
Bauer et al. Reusing system states by active learning algorithms
CN113157411A (en) Reliable configurable task system and device based on Celery
CN110222046B (en) List data processing method, device, server and storage medium
CN110888909B (en) Data statistical processing method and device for evaluation content
CN117389526A (en) Data offline export method and device, electronic equipment and storage medium
CN112181695A (en) Abnormal application processing method, device, server and storage medium
CN113495723B (en) Method, device and storage medium for calling functional component
CN112463626B (en) Memory leakage positioning method and device, computer equipment and storage medium
CN112764837B (en) Data reporting method, device, storage medium and terminal
CN116263717A (en) Order service processing method and device based on event
CN115269345A (en) Database testing method, device, equipment and storage medium
CN111045891B (en) Monitoring method, device, equipment and storage medium based on java multithreading
CN113656378A (en) Server management method, device and medium
CN112241141A (en) Hardware configuration method, system, device and medium of PLC control system
CN112988277A (en) Rule file loading method, device, server and medium
CN112765200A (en) Data query method and device based on Elasticissearch
US7269610B2 (en) System and method to observe user behavior and perform actions introspectable objects
CN113268266B (en) Multi-version coexistence management method, system and medium for applet rendering framework

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination