CN112597224A - Data export method, data export device, electronic equipment and medium - Google Patents

Data export method, data export device, electronic equipment and medium Download PDF

Info

Publication number
CN112597224A
CN112597224A CN202011522164.2A CN202011522164A CN112597224A CN 112597224 A CN112597224 A CN 112597224A CN 202011522164 A CN202011522164 A CN 202011522164A CN 112597224 A CN112597224 A CN 112597224A
Authority
CN
China
Prior art keywords
data
export
task
data export
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011522164.2A
Other languages
Chinese (zh)
Inventor
李泽锋
李平凉
李清意
禹龙晨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202011522164.2A priority Critical patent/CN112597224A/en
Publication of CN112597224A publication Critical patent/CN112597224A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present disclosure provides a data export method applied to a first server. The method comprises the following steps: receiving a call request for executing a first data export task; obtaining the export data volume of the first data export task according to the information of the calling request; determining a data export mode based on the export data amount of the first data export task, wherein the data export mode comprises the following steps: when the export data amount of the first data export task is larger than or equal to a first preset threshold value, adopting multithreading to concurrently export data; when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread; determining at least one data export file to export data into the at least one data export file. The disclosure also provides a data export method and device applied to the second server, electronic equipment and a storage medium. The data export method, the data export device, the electronic equipment and the storage medium can be used in the financial field or other fields.

Description

Data export method, data export device, electronic equipment and medium
Technical Field
The present disclosure relates to the field of finance, and more particularly, to a data export method, a data export apparatus, an electronic device, and a storage medium.
Background
As computer technology evolves, with the enormous data generation, data needs to be collected, stored, exported, and analyzed. By using the existing data export function, the user can obtain the required target data according to the selected data types and the screening conditions. When the scene with small data volume, simple service and small concurrency quantity is faced, the existing data export function can meet most data export requirements.
In the course of implementing the disclosed concept, the inventors found that there are at least the following problems in the prior art: when exporting large data, a large amount of system resources are occupied, so that the performance of a data exporting function is reduced, and the user experience is reduced.
Disclosure of Invention
In view of this, the embodiments of the present disclosure provide a data export method, an apparatus, an electronic device, and a storage medium, which can select different export manners according to the size of the data size of the export data.
One aspect of the disclosed embodiments provides a data export method applied to a first server. The data derivation method comprises the following steps: receiving a call request for executing a first data export task; obtaining the export data volume of the first data export task according to the information of the calling request; determining a data export mode based on the export data amount of the first data export task, wherein the data export mode comprises the following steps: when the export data amount of the first data export task is larger than or equal to a first preset threshold value, adopting multithreading to concurrently export data; when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread; determining at least one data export file to export data into the at least one data export file.
According to an embodiment of the present disclosure, the concurrently exporting data by using multiple threads includes: the first data export task is disassembled into N second data export tasks, and the export data volume of the second data export tasks is smaller than the first preset threshold; and executing N second data export tasks by adopting multiple threads, wherein N is an integer greater than or equal to 2.
According to an embodiment of the present disclosure, the disassembling the first data export task into N second data export tasks includes: setting a second preset threshold, wherein the second preset threshold is smaller than the first preset threshold; and distributing a part of the first data export task to one second data export task every other second preset threshold from the initial position of the first data export task to obtain N second data export tasks.
According to an embodiment of the present disclosure, said concurrently executing N second data derivation tasks using multiple threads includes: determining a first data export file; acquiring the initial position of export data of the first data export task in the first data export file; obtaining an export position in the first data export file every other second preset threshold from the starting position until obtaining N-1 export positions; writing export data for N of the second data export tasks to the first data export file based on the starting location and N-1 of the export locations.
According to an embodiment of the present disclosure, the concurrently executing N second data export tasks using multiple threads includes performing the following operations in a loop until all of the N second data export tasks are exported: determining a second data export file exported by the first data export task; if the available space of the second data export file is larger than or equal to the export data amount of the second data export task to be exported next, writing the second data export task into the second data export file; or if the available space of the second data export file is smaller than the export data amount of the second data export task to be exported next, creating a third data export file, and writing the second data export task into the third data export file.
According to an embodiment of the present disclosure, the determining at least one data export file comprises: if the first data export task exports data for the first time, a fourth data export file is created to export the data into the fourth data export file; or if the first data export task is not exporting data for the first time, exporting data to a fifth data export file, wherein the fifth data export file is a file from which data is exported last time by the first data export task.
Another aspect of the disclosed embodiments provides a data export method applied to a second server. The data export method comprises the following steps: acquiring a third data export task; determining a target server corresponding to the third data derivation task from at least one first server, wherein the determining comprises: inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type; issuing a call request to cause the target server to perform the third data export task.
According to an embodiment of the present disclosure, the third data export task is pre-joined to a message queue, and the acquiring the third data export task includes: obtaining the third data export task from the message queue for asynchronous processing.
According to an embodiment of the present disclosure, the issuing of the call request includes: inquiring a data service configuration table according to the data service type of the third data export task to obtain interface information of the target server, wherein the data service configuration table comprises the interface information of the at least one first server and the prestored corresponding relation; and sending out a calling request based on the interface information of the target server.
According to an embodiment of the present disclosure, before the querying the data service configuration table, the method further includes: setting interface information of the at least one first server in the data service configuration table; and setting the corresponding relation between the at least one first server and the data service type in the data service configuration table.
Another aspect of the disclosed embodiments provides a data export apparatus applied to a first server. The data exporting device comprises a receiving request module, a data inquiring module, a first determining module and a file determining module. The task obtaining module is used for obtaining a third data export task; the second determining module is configured to determine, from the at least one first server, a target server corresponding to the third data export task, where the second determining module includes: inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type; and the sending request module is used for sending a calling request so as to enable the target server to execute the third data export task.
Another aspect of the embodiments of the present disclosure provides a data exporting apparatus applied to a second server. The data exporting device comprises an acquiring task module, a second determining module and a sending request module. The task obtaining module is used for obtaining a third data export task; the second determining module is configured to determine, from the at least one first server, a target server corresponding to the third data export task, where the second determining module is configured to: inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type; and the sending request module is used for sending a calling request so as to enable the target server to execute the third data export task.
Another aspect of the disclosed embodiments provides an electronic device. The electronic device includes one or more memories, and one or more processors. The memory has stored thereon computer-executable instructions. The processor executes the instructions to implement the method as described above.
Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having stored thereon executable instructions, which when executed by a processor, cause the processor to perform the method as described above.
According to yet another aspect of an embodiment of the present disclosure, there is provided a computer program product comprising computer programs/instructions which, when executed by a processor, implement the method as described above.
One or more of the above-described embodiments may provide the following advantages or benefits: the problem that data export performance is reduced when a large data volume is exported can be at least partially solved, and data export is carried out by adopting a corresponding data export mode according to the size of the exported data volume, so that system resources are effectively utilized, and the data export performance is improved. The high concurrency scene of the data export task can be established for multiple users, the data export task is processed asynchronously, and the stability of the server is improved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates a system architecture for data export according to an embodiment of the present disclosure;
fig. 2 schematically shows a flow chart of a data export method applied to a second server according to an embodiment of the present disclosure;
FIG. 3 schematically shows a flow diagram for setting a data traffic configuration table according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a flow diagram of a second server issuing a call request, according to an embodiment of the disclosure;
FIG. 5 schematically shows a flow chart of a data export method applied to a first server according to an embodiment of the present disclosure;
FIG. 6 schematically illustrates a flow diagram for determining a data export file according to an embodiment of the present disclosure;
FIG. 7 schematically illustrates a flow diagram for exporting data concurrently with multiple threads, according to an embodiment of the present disclosure;
FIG. 8 schematically shows a flow diagram of a breakdown of a first data export task into N second data export tasks, according to an embodiment of the disclosure;
FIG. 9 schematically illustrates a flow diagram for concurrently executing N second data-export tasks using multiple threads, according to an embodiment of the disclosure;
FIG. 10 schematically illustrates a flow diagram for concurrently executing N second data-export tasks using multiple threads, according to another embodiment of the disclosure;
fig. 11 schematically shows a block diagram of a data exporting apparatus applied to a second server according to an embodiment of the present disclosure;
FIG. 12 schematically illustrates an application scenario diagram of a get task module according to an embodiment of the disclosure;
FIG. 13 schematically illustrates a flow of timing the storing of messages into a message queue according to an embodiment of the disclosure;
fig. 14 schematically shows a block diagram of a data derivation arrangement applied to a first server, in accordance with an embodiment of the present disclosure;
FIG. 15 schematically illustrates a flow of timing task message dequeue from a message queue and execution, according to an embodiment of the present disclosure.
FIG. 16 schematically shows a block diagram of a computer system suitable for implementing the data export method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides a data export method applied to a first server. The data export method comprises the following steps: receiving a call request for executing a first data export task; obtaining the export data volume of the first data export task according to the information of the calling request; determining a data export mode based on the export data amount of the first data export task, wherein the data export mode comprises the following steps: when the export data amount of the first data export task is larger than or equal to a first preset threshold value, adopting multithreading to concurrently export data; when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread; determining at least one data export file to export data into the at least one data export file.
The embodiment of the disclosure also provides a data export method applied to the second server. The data export method comprises the following steps: acquiring a third data export task; determining a target server corresponding to the third data derivation task from at least one first server, wherein the determining comprises: inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type; issuing a call request to cause the target server to perform the third data export task. Embodiments of the present disclosure also provide a data export apparatus, an electronic device, and a medium.
Fig. 1 schematically shows a system architecture of a data export method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios. For example, one embodiment of the present disclosure may be applied to the field of financial technology, and export data generated in financial transactions to facilitate analysis and processing of the data.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, 104, networks 105, 106, a task management server 107 and data traffic servers 108, 109, 110. The network 105 serves as a medium for providing a communication link between the terminal devices 101, 102, 103, 104 and the task management server 107, and the network 106 serves as a medium for providing a communication link between the task management server 107 and the data service servers 108, 109, 110. Networks 105 and 106 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may use the terminal devices 101, 102, 103, 104 to interact with the task management server 107 via the network 105 to send, receive messages to process data export tasks, and the like. The task management server 107 sends a call request to the data traffic servers 108, 109, 110 via the network 106. Various messaging client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103, and 104.
Terminal devices 101, 102, 103, and 104 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The task management server 107 may be a server that provides various services, such as a background management server (for example only) that provides support for websites, logged-in service systems, applications, and the like that users browse using the terminal devices 101, 102, 103, and 104. The backend management server may analyze and perform other processing on the received data such as the user data export request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.
The data service servers 108, 109, 110 may deploy specific data service systems, wherein the data service systems have specialized service personnel maintenance. The data service servers 108, 109, 110 may be servers for providing functions of storing, exporting, acquiring, or managing data, and execute the call request issued by the task management server 107.
It should be noted that the data export method applied to the second server provided by the embodiment of the present disclosure may be generally performed by the task management server 107. Accordingly, the data export device applied to the second server provided by the embodiment of the present disclosure may be generally disposed in the task management server 107. The data export method applied to the second server provided by the embodiment of the present disclosure may also be performed by a server or a server cluster different from the task management server 107. Accordingly, the data export device applied to the second server provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the task management server 107.
It should be noted that the data export method applied to the first server provided by the embodiment of the present disclosure may be generally performed by at least one of the data service servers 108, 109, and 110. Accordingly, the data export device applied to the first server provided by the embodiment of the present disclosure may be generally disposed in at least one of the data service servers 108, 109, and 110. The data export method applied to the first server provided by the embodiment of the present disclosure may also be performed by a server or a server cluster different from the data service servers 108, 109, and 110. Correspondingly, the data export device applied to the first server provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the data service servers 108, 109, and 110.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks and servers, depending on the application.
The following describes an exemplary embodiment of the present disclosure with reference to the system architecture 100 of fig. 1. Among other things, the task management server 107 can be used to process data export tasks for users. The data that the user wants to export can be managed by any one of the data service servers 108, 109, 110, wherein the corresponding data can be stored in the data service servers 108, 109, 110, or can be acquired from other databases by the data service servers 108, 109, 110.
Fig. 2 schematically shows a flowchart of a data export method applied to a second server according to an embodiment of the present disclosure. Wherein the second server may be the task management server 107.
In operation S210, a third data export task is acquired;
according to an embodiment of the present disclosure, referring to fig. 1, a user may create a data export task through a terminal device 101. For example, when a user wants to download consumption schedule in a bank account, the user first screens conditions such as account, date, amount interval, income or expense, then clicks an export button on a bank webpage to newly create a download task, and finally obtains a data export task created by the user and related to the consumption schedule of the bank account by the task management server 107.
According to an embodiment of the present disclosure, a third data export task is pre-joined to the message queue, and the acquiring the third data export task includes: obtaining the third data export task from the message queue for asynchronous processing.
Referring to fig. 1, first, for example, after a user creates a data export task through a terminal device 101, a task management server 107 receives a record of the data export task, where the data export task includes information such as screening conditions and types of data services that the user wants to download. It should be appreciated that the task management server 107 may process a plurality of tasks created by one user or a plurality of tasks created by a plurality of users, for example, a task list including a plurality of data export tasks may be formed.
Then, for example, the task management server 107 is made to periodically traverse the tasks in the query task list whose states are not completed by execution, and store the derived tasks that are not completed by execution into the message queue.
Finally, the task management server 107 may be configured to fetch the data export task from the message queue at intervals, and asynchronously process the data export task to achieve the purpose of acquiring data.
By using the data export method of the embodiment of the disclosure, after a user creates a data export task, the task is not processed immediately to avoid occupying current resources, but the data export task is put into a message queue, and the data export task is acquired from the message queue at other time and is processed asynchronously. Thereby enabling separation of the task of creating data exports and the specific execution of data exports. The user can do other operations in the process of waiting for the completion of the file downloading, so that the user experience is improved.
In operation S220, a target server corresponding to the third data export task is determined from the at least one first server, where the method includes: and inquiring a target server corresponding to the third data export task according to the pre-stored corresponding relation between the at least one first server and the data service type.
The following describes in detail a process of querying the target server corresponding to the third data export task in operation S220 with reference to fig. 3.
Fig. 3 schematically shows a flow chart of setting a data traffic configuration table according to an embodiment of the present disclosure.
As shown in fig. 3, the data export method according to the embodiment of the present disclosure further includes setting a data service configuration table, which may specifically include operations S310 to S320.
In operation S310, interface information of the at least one first server in the data service configuration table is set.
In operation S320, a correspondence relationship between the at least one first server and a data service category in the data service configuration table is set.
According to an embodiment of the present disclosure, referring to fig. 1, for example, a data service system for managing the pipelining of bank accounts is deployed in the data service server 108, a data service system for managing periodic financing is deployed in the data service server 109, and a data service system for managing noble metal transactions is deployed in the data service server 110. In addition, a data service configuration table may be provided in the task management server 107.
First, since the data service servers 108, 109, and 110 are respectively deployed with different types of data service systems, the corresponding relationship between each data service server 108, 109, and 110 and the data service type may be set in the data service configuration table.
Secondly, the service personnel can set the corresponding interface information in the data service configuration table according to different interface specifications of the data service servers 108, 109 and 110. Based on the interface information, the task management server 107 can generally call the data service servers 108, 109, and 110 to perform a data export function to implement a method of acquiring data according to a specific data service.
According to an embodiment of the present disclosure, operations S310 to S320 may be performed to set the data service configuration table before operation S210. After operation S210, the target server corresponding to the third data export task may be queried using the interface information and the corresponding relationship in the pre-stored data configuration table, and then operation S220 is performed.
In operation S230, a call request is issued to cause the target server to perform the third data export task.
The flow of issuing the call request in operation S230 is described in detail below with reference to fig. 4.
Fig. 4 schematically shows a flow chart of the second server issuing the invocation request according to an embodiment of the present disclosure.
As shown in fig. 4, the issuing of the call request by the second server may further include operations S410 to S420.
In operation S410, a data service configuration table is queried according to the data service type of the third data export task to obtain the interface information of the target server, where the data service configuration table includes the interface information of the at least one first server and the pre-stored correspondence relationship.
In operation S420, a call request is issued based on the interface information of the target server.
According to an embodiment of the present disclosure, referring to fig. 1, for example, after the task management server 107 extracts a data export task created by the user using the terminal device 101 from the message queue, first, the data type of the data export task is read as bank account details. Next, the task management server 107 queries the data service configuration table, and obtains the bank account flow details according to the pre-configured corresponding relationship, which are managed by the data service system deployed in the data service server 108. Then, according to the interface information in the data service configuration table, the task management server 107 uses a method called in a generalization manner to make the data service server 108 execute the data export task.
By using the data export method of the embodiment of the disclosure, a plurality of data service systems can share one set of data export source codes. For example, in the prior art, each data service server has different interface specifications, so that different data export service logics exist, and development needs to be performed on each service logic to meet data export requirements of users. The data export method of the embodiment of the disclosure can integrate and asynchronously process the data export tasks created by different users through the task management server 107, so that the data service systems with different service logics share one set of data export services, thereby effectively reducing the development cost.
Fig. 5 schematically shows a flow chart of a data export method applied to a first server according to an embodiment of the present disclosure. The first server may be any one of the data service servers 108, 109, and 110 in fig. 1.
As shown in fig. 5, the data export method applied to the first server may include operations S510 to S540.
In operation S510, a call request to perform a first data export task is received.
In operation S520, an export data amount of the first data export task is obtained according to the information of the call request.
According to the embodiment of the disclosure, for example, the task management server 107 receives the data export task related to the bank account details, and sends the invocation request to the data service server 108 by using a generalized invocation mode. The information of the call request includes parameters such as account name, date, amount interval, income and the like. Then, the data service server 108 queries the data that the user needs to export by calling the requested information.
In operation S530, a data export manner is determined based on the export data amount of the first data export task, which may include operations S531 to S532.
In operation S531, when the derived data amount of the first data deriving task is greater than or equal to a first preset threshold, concurrently deriving data using multiple threads.
In operation S532, when the derived data amount of the first data deriving task is smaller than a first preset threshold, the data is derived using a single thread.
According to the disclosed embodiment, the derived data amount is compared with a preset download threshold, and data is derived in a corresponding data deriving mode based on the comparison result. For example, the preset threshold may be a maximum download amount of data set by a back-office worker of the bank according to the performance of the server.
For example, when the data amount of the user export task is greater than or equal to the preset maximum data download amount, the data export task can be executed in a multi-thread manner in a concurrent manner. When the data volume of the user exported task is smaller than the preset maximum data download volume, the data can be exported in a single thread mode.
In operation S540, at least one data export file is determined to export data into the at least one data export file.
The process of exporting data to a file in operation S540 is described in detail below with reference to fig. 6.
FIG. 6 schematically shows a flow diagram for determining a data export file according to an embodiment of the present disclosure.
As shown in fig. 6, determining the data export file may include operations S610 to S620.
In operation S610, if the first data export task exports data for the first time, a fourth data export file is created to export data into the fourth data export file.
In operation S620, if the first data export task is not exporting data for the first time, exporting data to a fifth data export file, wherein the fifth data export file is a file from which data was exported last time by the first data export task.
With the embodiments of the present disclosure, the amount of data for a user-derived task is compared to a preset threshold. If the data volume is less than the set value, the data can be exported in a single thread mode, and because the data volume is small, excessive resources cannot be occupied. If the value is larger than the set value, data can be exported in a multithread mode. Therefore, the problem of page blocking caused by the fact that a user creates a large data volume export task in the prior art can be effectively solved, data export performance can be improved, and better experience is provided for the user.
FIG. 7 schematically shows a flow diagram for exporting data concurrently with multiple threads, according to an embodiment of the present disclosure.
As shown in fig. 7, concurrently exporting data using multiple threads in operation S531 according to an embodiment of the present disclosure may include operations S710 to S720.
In operation S710, the first data export task is decomposed into N second data export tasks, where the export data amount of the second data export tasks is smaller than a preset threshold.
In operation S720, N second data derivation tasks are concurrently executed using multiple threads, where N is an integer greater than or equal to 2.
According to the embodiment of the disclosure, when the data export task created by the user corresponds to a large data volume, for example, the user wants to download the flow details in the bank account, and after conditions such as the account, the date, the money interval, the income or the expense are set, the flow details data has 10 ten thousand. For example, if the preset threshold is 1 ten thousand, the derived target flow detail data is greater than the preset threshold. Then, the tasks including 10 ten thousand pieces of data are disassembled, a plurality of export tasks with the data size smaller than a preset threshold value are disassembled, and then the tasks with a plurality of small data sizes are executed concurrently in a multithreading mode.
By using the data export method provided by the embodiment of the disclosure, the large data download task is disassembled into a plurality of small data export tasks, and then the small data export tasks are exported in batches, so that the data export performance can be effectively improved, and the export problem of large data files is solved.
Fig. 8 schematically illustrates a flow diagram for disaggregating a first data-export task into N second data-export tasks according to an embodiment of the present disclosure.
As shown in fig. 8, the disassembling of the first data export task into N second data export tasks may include operations S810 to S820 according to an embodiment of the present disclosure.
In operation S810, a second preset threshold is set, and the second preset threshold is smaller than the first preset threshold.
In operation S820, a portion of the first data export task is allocated to one second data export task every second preset threshold from the start position of the first data export task, so as to obtain N second data export tasks.
According to the embodiment of the disclosure, for example, 10 pieces of serial detail data of a bank account derived by a user are provided, while 1 piece of the first preset threshold is provided, and 5000 pieces of the second preset threshold can be provided. And from the initial position of the 10 ten thousand pieces of data, distributing a part of the 10 ten thousand pieces of data to a small-data-volume export task (such as 0-5000, 5001-10000 … …) every 5000 pieces of data until the 10 ten thousand pieces of data are disassembled, so as to obtain 20 small-data-volume export tasks.
It should be noted that the first preset threshold and the second preset threshold may be flexibly set according to factors such as the processing performance of the server or the service condition of the current data exporting function. For example, the second preset threshold is 7000, and when the number of the last task is not 7000 from the starting position to the ending position of the 10 ten thousand pieces of data, the 10 th ten thousand pieces of data may be set as a disassembly ending condition, that is, the last task includes 98001 to 100000 pieces of data. The present disclosure is not particularly limited thereto.
Fig. 9 schematically illustrates a flow diagram for concurrently executing N second data-export tasks using multiple threads, according to an embodiment of the disclosure.
As shown in fig. 9, concurrently executing N second data export tasks using multiple threads according to an embodiment of the present disclosure may include operations S910 to S940.
In operation S910, a first data export file is determined. The first data export file is used for receiving data writing in the N second data export tasks.
According to the embodiment of the disclosure, for example, for 10 ten thousand pieces of data exported by a user, a data export task with a small data size is disassembled every 5000 pieces of data, and after a task with a small data size of 20 is obtained, a plurality of threads are used for concurrently executing the 20 tasks. First, it is determined whether the 10 ten thousand pieces of data were first exported, such as querying an export log, querying a user to export a task record, or scanning whether an export file already exists. Next, for example, operations S610 to S620 may be performed, and if the 10 ten thousand pieces of data are first exported, a data export file is created, and if the 10 ten thousand pieces of data are not first exported, a previously created source export file is found, and the source export file is used as the first data export file. And finally, determining a first data export file according to the judgment result.
In operation S920, a start position of exporting data into the first data export file is obtained.
According to the embodiment of the present disclosure, if a user exports 10 ten thousand pieces of data for the first export and creates a new export file, the start position of writing the 10 ten thousand pieces of data in the export file is obtained first. For example, the export file format is an excel format, and the start position is the first line of the first table in the excel.
According to another embodiment of the present disclosure, if the user does not export 10 ten thousand pieces of data for the first time, the 10 ten thousand pieces of data are additionally written in the existing source export file. At this time, the start position of the additional write in the source file needs to be acquired.
In operation S930, one export position in the first data export file is obtained every second preset threshold from the start position until N-1 export positions are obtained
In operation S940, export data of the N second data export tasks are written to the first data export file based on the start position and the N-1 export positions.
For example, in the case of a user's demand for exporting 10 ten thousand pieces of data, where a data export job of a small data amount is disassembled for every 5000 pieces of data, and a job of 20 small data amounts is obtained, after the start position of the written data is determined, an export position is first obtained for every 5000 pieces from the start position until 19 export positions are obtained. The starting location and the 19 derived locations are in one-to-one correspondence with the 20 disassembled tasks (e.g., locations in the file corresponding to the 1 st, 5001 st, and 10001 st pieces of data … …). And finally, writing 20 data export tasks into the file in parallel according to the starting position and the 19 export positions.
According to embodiments of the present disclosure, if an export file already exists and the source file is an empty file, data is written directly to the file after the start location and the N-1 export locations are obtained. If part of data already exists in the source file, after the starting position and the N-1 export positions in the source file are obtained, the existing data can be covered, or the existing part of data can be compared with the exported data, and the export of the existing data is not performed. The file export method is not limited, and can be set according to actual conditions, and the finally exported file comprises all data required by a user and does not contain repeated or redundant data.
According to the embodiment of the disclosure, after the positions of the N second export tasks written in the first data export task are determined, the N second export tasks can be exported in batch in parallel.
Fig. 10 schematically illustrates a flow diagram for concurrently executing N second data-export tasks using multiple threads, according to another embodiment of the disclosure.
As shown in fig. 10, according to another embodiment of the present disclosure, concurrently executing N second data export tasks using multiple threads may further include performing operations S1010 to S1030 in a loop until all of the N second data export tasks are exported.
In operation S1010, a second data export file exported by the current first data export task is determined.
In operation S1020, if the available space of the second data export file is greater than or equal to the export data amount of the second data export task to be exported next, the second data export task is written into the second data export file.
In operation S1030, if the available space of the second data export file is smaller than the export data amount of the second data export task to be exported next, a third data export file is created and the second data export task is written in the third data export file.
According to the embodiment of the disclosure, for example, 10 ten thousand pieces of data are disassembled to obtain 20 tasks with small data size, and the 20 tasks correspond to 1-5000 pieces of data and 5001-10000 pieces of data 10000 … … 95001-100000 pieces of data respectively.
First, an export file for each task to be exported next is determined. Specifically, for example, when a first task corresponding to 1 to 5000 pieces of data is executed, whether an export file already exists locally is scanned to determine whether 10 ten thousand pieces of data are exported for the first time. If the export file does not exist, namely the export is carried out for the first time, an export file (hereinafter referred to as file A) is created, and the first task is written. When a second task corresponding to 5001-10000 pieces of data is executed, whether an export file already exists in the local is scanned, at the moment, a file A created by a first task is scanned, and the file A is used as a second data export file exported by the current first data export task. For example, when an export file is specified for the last task of 95001 to 100000 pieces of data, if a plurality of export files are scanned, the export file that has been created last can be specified as the second data export file according to the creation time of the export file.
Then, after the second data export file is determined, the export data amount of the next task to be exported is compared with the available space of the second data export file. For example, a created export file A may hold 12000 pieces of data, and then a task containing 1 to 5000 pieces of data is compared with the available space of the file A, and if the comparison shows that the file A can allow 5000 pieces of data to be written, then a task containing 1 to 5000 pieces of data is written into the file A.
File a may then also hold 10000 pieces of data. When a task containing 5001-10000 pieces of data is executed, the data size of 5000 pieces of data is compared with the available space of the file A, and if the file A is judged to allow 5000 pieces of data to be written, the task containing 5001-10000 pieces of data is written into the file A.
File a may then also hold 2000 pieces of data. When executing a task containing 10001-15000 data, scanning out the file A, and using the file A as an export file to be written next. The data size of 5000 pieces is compared with the available space of file a. And if the available space of the file A is judged to be less than 5000 pieces of export data, creating a new export file B (namely, a third data export file), and writing data containing 10001-15000 pieces of data into the newly created export file B, wherein the export file B can contain 12000 pieces of data.
And then, the export file B can also contain 7000 pieces of data, when a task containing 15001-20000 pieces of data is executed, the file A and the file B are scanned, and the file B which is created later is determined as an export file to be written next according to the sequence of creation time. The amount of data containing 5000 pieces is compared with the available space of file B and then written to file B.
Finally, referring to the above steps, operations S1010 to S1030 are continuously and cyclically performed for the remaining 16 tasks until all of the 20 data export tasks are completed.
It should be noted that the above quantization standard using the number of pieces of data as the size of the export data is only one embodiment of the present disclosure, and the present disclosure does not specifically limit the quantization unit of the data amount and the format of the export file.
According to another embodiment of the present disclosure, concurrently performing the N second data derivation tasks using multiple threads may further include: and writing each second data export task into a single data export file so as to obtain N data export files. And after the N second data export tasks are executed in parallel, merging the generated N data files to obtain a file containing all export data.
Fig. 11 schematically shows a block diagram of a data exporting apparatus applied to a second server according to an embodiment of the present disclosure.
As shown in fig. 11, the data exporting apparatus 1100 includes a get task module 1110, a second determination module 1120, and a send request module 1130. The data export apparatus 1100 may be used to implement the data export method described with reference to fig. 2 to 4.
Specifically, the get task module 1110 may perform operation S210, for example, for obtaining a third data export task.
The second determining module 1120 may perform operation S220, for example, to determine a target server corresponding to the third data derivation task from at least one first server, where the determining includes: and inquiring a target server corresponding to the third data export task according to the pre-stored corresponding relation between the at least one first server and the data service type.
The send request module 1130 may perform, for example, operation S230 for issuing a call request to cause the target server to perform the third data export task.
According to an embodiment of the present disclosure, the data export apparatus 1100 may further include a configuration table module, for example, where the configuration table module may further perform operations S310 to S320, which are not described herein.
According to an embodiment of the present disclosure, the second determining module 1120 may further perform, for example, operation S410, and transmit the query result to the transmission requesting module 1130, and then the transmission requesting module 1130 may perform operation S420.
FIG. 12 schematically illustrates an application scenario diagram of the get task module 1110 according to an embodiment of the present disclosure.
As shown in fig. 12, the application scenario of the get task module 1110 according to an embodiment of the present disclosure may include operations S1210 to S1250.
In operation S1210, after a user creates an export task, an asynchronous data export task is acquired.
In operation S1220, parameters (e.g., data traffic category, data filtering condition, etc.) carried in the data export task created by the user are encapsulated for storing in the message queue.
In operation S1230, a plurality of messages are included in the message queue, and each message represents a data export task placed by the get task module 1110.
In operation S1240, a message is fetched from the message queue.
In operation S1250, the message fetched from the message queue, i.e., the data export task, is processed. And acquiring a business exporting method configured by business personnel according to the data business configuration table so as to realize data export.
According to the embodiment of the disclosure, the method of polling, timing or random time can be adopted to store or take out the message in the message queue, so as to separate the task of creating data export by the user from the data export, thereby realizing asynchronous processing of the data export task.
Fig. 13 schematically illustrates a flow of timing the storing of messages into a message queue according to an embodiment of the disclosure.
As shown in FIG. 13, timing the storing of messages in the message queue may include operations S1310-S1370. Which may be performed by the task management server 107.
In operation S1310, referring to fig. 1, for example, the timing operation may be performed by using the task management server 107, and a parameter of the timing operation may be set in advance, such as performing the timing operation every 5 seconds.
In operation S1320, each data export task in the task list is periodically traversed.
In operation S1330, it is determined whether the execution status of each data export task is in execution, if so, operation S1340 is performed, and if not, operation S1360 is performed.
In operation S1340, the execution time of the executing data export task is obtained, and it is determined whether the execution is overtime, if yes, operation S1350 is performed, and if not, operation S1370 is performed.
In operation S1350, if the execution of the data export task is timed out, the execution status of the data export task is updated to fail. For a data export task that fails to perform, the results may be fed back to the user who created the data export task, created again by the user, or contacted by business personnel, for example.
In operation S1360, if the data export task is not executed, the data export task is a to-be-executed task, and is added to the message queue.
In operation S1370, the timing operation is ended. In addition, for example, after the timing operation is ended, the execution result may be fed back.
Fig. 14 schematically shows a block diagram of a data derivation apparatus applied to a first server according to an embodiment of the present disclosure.
As shown in FIG. 14, the data export apparatus 1400 includes a get task module 1410, a query data module 1420, a first determination module 1430, and a file determination module 1440. The data exporting apparatus 1400 may be used to implement the data exporting method described with reference to fig. 5 to 10.
Specifically, the receive request module 1410 may perform, for example, operation S510 for receiving a call request to execute a first data export task.
The query data module 1420 may, for example, perform operation S520, for obtaining the export data amount of the first data export task according to the information of the call request.
The first determining module 1430 may perform operation S530, for example, to determine a data export manner based on the export data amount of the first data export task, where the data export manner includes concurrently exporting data in multiple threads when the export data amount of the first data export task is greater than or equal to a first preset threshold; when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread;
the file determining module 1440 may perform, for example, operation S540 for determining at least one data export file to export data into the at least one data export file.
FIG. 15 schematically illustrates a flow of timing task message dequeue from a message queue and execution, according to an embodiment of the present disclosure.
As shown in fig. 15, timing the storing of the message in the message queue may include operations S1501 to S1515. Among them, operations S1501 to S1506 and S1515 may be performed by the task management server 107, and operations S1507 to S1514 may be performed by any one of the data service servers 108, 109 or 110.
In operation S1501, the task management server 107 is caused to fetch one or more data export tasks from the message queue every 5 seconds, for example.
In operation S1502, a null value check is performed on each task message, whether the retrieved task message includes information required for execution is determined, and if the message is a null value without any information, the task cannot be executed, and then the process is terminated. If not, operation S1503 is performed.
In operation S1503, for example, each time a plurality of task messages are fetched from the message queue, the plurality of task messages are distributed to a plurality of threads, and the data export task is asynchronously executed.
In operation S1504, the execution state is modified in the task list to be in execution for the data export task started to be executed in operation S1503, and the execution time is updated.
In operation S1505, a task is derived according to the retrieved data, and interface information of the target server is obtained, which may refer to operation S410, which is not described herein again.
In operation S1506, a call request is issued by using a generalized call method according to the acquired interface information, and reference may be made to operation S420.
According to the embodiment of the disclosure, specific service class information configured by service personnel according to the data service server interface specification can be read, and then a data export task method is called in a generalization mode according to the specific service class information.
According to the embodiment of the disclosure, for example, a service development module may be configured in the task management server, and the service development module is shared by all data service servers. In the service development module, each service developer maintaining the data service server can adapt to the corresponding sub-service, and can create a subclass (for example, interface information of the data service server) containing a specific method for acquiring service data in the sub-service, so as to acquire data according to the specific method for acquiring the service.
In operation S1507, a data export task is performed. For example, target data is searched according to the screening condition set by the user in the data export task.
In operation S1508, it is determined whether to concurrently execute the task according to the size of the retrieved export data and a preset export threshold, and a file of the export data is generated according to the file generation manner and the query result. If the tasks are concurrently executed, operation S1509 is performed, and if the tasks are not concurrently executed, operation S1511 is performed.
According to the embodiment of the disclosure, if the query result is null, a null file is generated. For example, data which needs to be exported by a user does not exist, after the data is connected to a data source, the data cannot be queried, a query result is returned to be null, a null file is generated for the user, and the data export is finished, so that the data export is facilitated to be continuously executed according to a pre-designed logic.
In operation S1509, the data export task is disassembled into a plurality of data export tasks of small data size.
In operation S1510, derived data is concurrently acquired from a start position of the derived data according to the parsed data derivation tasks of the plurality of small data amounts.
In operation S1511, if the data is not concurrently executed, the single thread executes the data export task, obtains the target data from the data source by paging, and writes the target data into an export file.
In operation S1512, since each file has a maximum download number limit, multiple files should be created when the available space of the file is insufficient to support a small amount of data for a data export task to export data completely.
According to the embodiment of the disclosure, before exporting data, whether the data is exported for the first time or not is judged, so that the export file is determined. If it is the first export, a new file is created and the data is exported. If the data is not first exported, the data is appended directly under the source file. If the additional data exceeds the available space of the file, a new file is created.
In operation S1513, after all the data export files are successfully created, all the files are compressed and uploaded to the designated download location, and the download address is sent to the user, so that the user can obtain the export data.
In operation S1514, after the file upload is successful, the local compressed file and the source file are deleted.
According to the embodiment of the disclosure, for example, a data export file can be created locally, after data export is completed, the file is compressed, and the compressed file is uploaded to a cloud server, so that a user can obtain data from the cloud server. And after the file is uploaded successfully, deleting the compressed file and the source file. And returns information of completion of the task processing to the task management server 107.
In operation S1515, the task information is updated. For example, a successful task may be deleted from the task list, or the execution state may be changed to be successful, so as to avoid repeated processing.
By utilizing the data export device disclosed by the embodiment of the disclosure, the unfinished and newly-built tasks are asynchronously added to the message queue through timing scanning, then the tasks in the queue are taken out through timing scanning, and the corresponding tasks are executed to realize the file downloading function. All data export services share one set of codes, and centralized maintenance is facilitated. Meanwhile, in the face of the export problem of the large data files, the data volume of the downloaded files is judged in advance, if the data volume exceeds the maximum downloading quantity of a single time, the large data volume downloading task can be disassembled into a plurality of small data volume exporting tasks according to the index position, then a plurality of data files are exported in a concurrent mode, and the data export performance is improved.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, the modules in the data export apparatus 1100 or the data export apparatus 1400 may be combined and implemented in one module, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the modules in the data export apparatus 1100 or the data export apparatus 1400 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware. Alternatively, at least one of the modules in the data exporting apparatus 1100 or the data exporting apparatus 1400 may be at least partially implemented as a computer program module, which when executed, may perform the corresponding function.
FIG. 16 schematically shows a block diagram of a computer system suitable for implementing the data export method according to an embodiment of the present disclosure. The computer system illustrated in FIG. 16 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 16, a computer system 1600 according to an embodiment of the present disclosure includes a processor 1601 that can perform various suitable actions and processes according to a program stored in a Read Only Memory (ROM)1602 or a program loaded from a storage portion 1608 into a Random Access Memory (RAM) 1603. Processor 1601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or related chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 1601 may also include on-board memory for caching purposes. Processor 1601 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the present disclosure.
In the RAM 1603, various programs and data necessary for the operation of the system 1600 are stored. The processor 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. Processor 1601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1602 and/or RAM 1603. It is to be noted that the program may also be stored in one or more memories other than the ROM 1602 and the RAM 1603. The processor 1601 can also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in one or more memories.
In accordance with an embodiment of the present disclosure, the system 1600 may also include an input/output (I/O) interface 1605, the input/output (I/O) interface 1605 also being connected to the bus 1604. The system 1600 may also include one or more of the following components connected to the I/O interface 1605: an input portion 1606 including a keyboard, a mouse, and the like; an output portion 1607 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 1608 including a hard disk and the like; and a communication section 1609 including a network interface card such as a LAN card, a modem, or the like. The communication section 1609 performs communication processing via a network such as the internet. The driver 1610 is also connected to the I/O interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1610 as necessary, so that a computer program read out therefrom is mounted in the storage portion 1608 as necessary.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the data deriving method provided by the embodiments of the present disclosure.
The computer program, when executed by the processor 1601, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure. In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, downloaded and installed via the communication part 1609, and/or installed from the removable medium 1611. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include one or more memories other than ROM 1602 and/or RAM 1603 and/or ROM 1602 and RAM 1603 described above.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims (14)

1. A data export method is applied to a first server, and comprises the following steps:
receiving a call request for executing a first data export task;
obtaining the export data volume of the first data export task according to the information of the calling request;
determining a data export mode based on the export data amount of the first data export task, wherein the data export mode comprises the following steps:
when the export data amount of the first data export task is larger than or equal to a first preset threshold value, adopting multithreading to concurrently export data;
when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread;
determining at least one data export file to export data into the at least one data export file.
2. A method of data export as claimed in claim 1, wherein the concurrently exporting data using multiple threads comprises:
the first data export task is disassembled into N second data export tasks, and the export data volume of the second data export tasks is smaller than the first preset threshold;
and executing N second data export tasks by adopting multiple threads, wherein N is an integer greater than or equal to 2.
3. The data export method of claim 2, wherein the disaggregation of the first data export task into N second data export tasks comprises:
setting a second preset threshold, wherein the second preset threshold is smaller than the first preset threshold;
and distributing a part of the first data export task to one second data export task every other second preset threshold from the initial position of the first data export task to obtain N second data export tasks.
4. The data export method of claim 2, wherein the concurrently executing N of the second data export tasks using multiple threads comprises:
determining a first data export file, wherein the at least one data export file comprises the first data export file;
acquiring a starting position for exporting data into the first data export file;
obtaining an export position in the first data export file every other second preset threshold from the starting position until obtaining N-1 export positions;
writing export data for N of the second data export tasks to the first data export file based on the starting location and N-1 of the export locations.
5. A data export method as claimed in claim 2, wherein said concurrently executing N of said second data export tasks using multiple threads comprises looping through the following until all N of said second data export tasks are exported:
determining a second data export file currently exported by the second data export task, wherein the at least one data export file comprises the second data export file;
if the available space of the second data export file is larger than or equal to the export data amount of the second data export task to be exported next, writing the second data export task into the second data export file; or
And if the available space of the second data export file is smaller than the export data amount of the second data export task to be exported next, creating a third data export file, and writing the second data export task into the third data export file, wherein the at least one data export file comprises the third data export file.
6. A data export method as claimed in claim 1, said determining at least one data export file comprising:
if the first data export task exports data for the first time, a fourth data export file is created to export the data into the fourth data export file; or
And if the first data export task is not exporting data for the first time, exporting the data to a fifth data export file, wherein the fifth data export file is a file from which the data is exported last time by the first data export task.
7. A data export method is applied to a second server and comprises the following steps:
acquiring a third data export task;
determining a target server corresponding to the third data derivation task from at least one first server, wherein the determining comprises:
inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type;
issuing a call request to cause the target server to perform the third data export task.
8. The data export method of claim 7 wherein the third data export task is pre-enqueued in a message queue, the obtaining a third data export task comprising:
obtaining the third data export task from the message queue for asynchronous processing.
9. The data export method of claim 7, the issuing a call request comprising:
inquiring a data service configuration table according to the data service type of the third data export task to obtain interface information of the target server, wherein the data service configuration table comprises the interface information of the at least one first server and the prestored corresponding relation;
and sending out a calling request based on the interface information of the target server.
10. A data derivation method as claimed in claim 9, further comprising, prior to said querying a data traffic configuration table:
setting interface information of the at least one first server in the data service configuration table; and
and setting the corresponding relation between the at least one first server and the data service type in the data service configuration table.
11. A data export device applied to a first server comprises:
the receiving request module is used for receiving a calling request for executing a first data export task;
the query data module is used for acquiring the export data volume of the first data export task according to the information of the call request;
a first determining module, configured to determine a data export manner based on an export data amount of the first data export task, where the first determining module includes:
when the export data amount of the first data export task is larger than or equal to a first preset threshold value, adopting multithreading to concurrently export data;
when the export data volume of the first data export task is smaller than a first preset threshold value, exporting data by adopting a single thread;
a file determination module to determine at least one data export file to export data into the at least one data export file.
12. A data export device is applied to a second server and comprises:
the task obtaining module is used for obtaining a third data export task;
a second determining module, configured to determine, from the at least one first server, a target server corresponding to the third data export task, where the second determining module includes:
inquiring a target server corresponding to the third data export task according to a pre-stored corresponding relation between the at least one first server and the data service type;
and the sending request module is used for sending a calling request so as to enable the target server to execute the third data export task.
13. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-6 or any of claims 7-10.
14. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 6 or any one of claims 7 to 10.
CN202011522164.2A 2020-12-21 2020-12-21 Data export method, data export device, electronic equipment and medium Pending CN112597224A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011522164.2A CN112597224A (en) 2020-12-21 2020-12-21 Data export method, data export device, electronic equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011522164.2A CN112597224A (en) 2020-12-21 2020-12-21 Data export method, data export device, electronic equipment and medium

Publications (1)

Publication Number Publication Date
CN112597224A true CN112597224A (en) 2021-04-02

Family

ID=75199926

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011522164.2A Pending CN112597224A (en) 2020-12-21 2020-12-21 Data export method, data export device, electronic equipment and medium

Country Status (1)

Country Link
CN (1) CN112597224A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116192816A (en) * 2022-06-01 2023-05-30 中兴通讯股份有限公司 Data transmission method, transmission server, reception server, and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101996067A (en) * 2009-08-19 2011-03-30 阿里巴巴集团控股有限公司 Data export method and device
CN111949402A (en) * 2020-08-05 2020-11-17 中国建设银行股份有限公司 Database request processing method and device, computer equipment and storage medium

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101996067A (en) * 2009-08-19 2011-03-30 阿里巴巴集团控股有限公司 Data export method and device
CN111949402A (en) * 2020-08-05 2020-11-17 中国建设银行股份有限公司 Database request processing method and device, computer equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116192816A (en) * 2022-06-01 2023-05-30 中兴通讯股份有限公司 Data transmission method, transmission server, reception server, and storage medium
WO2023231897A1 (en) * 2022-06-01 2023-12-07 中兴通讯股份有限公司 Data transmission methods, sending server, receiving server and storage medium

Similar Documents

Publication Publication Date Title
CN107729139B (en) Method and device for concurrently acquiring resources
CN110310034B (en) Service arrangement and business flow processing method and device applied to SaaS
US20220327125A1 (en) Query scheduling based on a query-resource allocation and resource availability
US20190310977A1 (en) Bucket data distribution for exporting data to worker nodes
US10698897B2 (en) Executing a distributed execution model with untrusted commands
US20190272271A1 (en) Assigning processing tasks in a data intake and query system
US20180089324A1 (en) Dynamic resource allocation for real-time search
CN111258565A (en) Method, system, server and storage medium for generating small program
CN111831461A (en) Method and device for processing business process
US11182144B2 (en) Preventing database package updates to fail customer requests and cause data corruptions
CN111427899A (en) Method, device, equipment and computer readable medium for storing file
CN112597224A (en) Data export method, data export device, electronic equipment and medium
US10893015B2 (en) Priority topic messaging
CN116431290A (en) Job scheduling method, apparatus, device, medium and program product
US11658930B2 (en) Personalized dashboard chart for email subscriptions
CN113064987B (en) Data processing method, apparatus, electronic device, medium, and program product
CN114168607A (en) Global serial number generation method, device, equipment, medium and product
CN112988806A (en) Data processing method and device
CN115484149B (en) Network switching method, network switching device, electronic equipment and storage medium
CN113259431B (en) Service calling method, service calling device, electronic device and storage medium
CN114640585B (en) Resource updating method and device, electronic equipment and storage medium
CN116257375A (en) Kafka data automatic stream processing method and device
US20240220237A1 (en) Smart grouping of code packages
CN118245176A (en) Escrow accounting task scheduling method and device
US11868382B1 (en) Software utility toolbox for client devices

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination