WO2021238796A1 - 下载任务异步处理方法、装置、系统和存储介质 - Google Patents

下载任务异步处理方法、装置、系统和存储介质 Download PDF

Info

Publication number
WO2021238796A1
WO2021238796A1 PCT/CN2021/095194 CN2021095194W WO2021238796A1 WO 2021238796 A1 WO2021238796 A1 WO 2021238796A1 CN 2021095194 W CN2021095194 W CN 2021095194W WO 2021238796 A1 WO2021238796 A1 WO 2021238796A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
download
target
tasks
asynchronous
Prior art date
Application number
PCT/CN2021/095194
Other languages
English (en)
French (fr)
Inventor
冀怀远
冯驰
胥磊
Original Assignee
苏宁易购集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏宁易购集团股份有限公司 filed Critical 苏宁易购集团股份有限公司
Publication of WO2021238796A1 publication Critical patent/WO2021238796A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Definitions

  • This application relates to the field of software development, in particular to a method, device, system and storage medium for asynchronous processing of download tasks.
  • the download function will inevitably be involved when developing business system functions.
  • synchronous downloading will be used to process download requests from the client; for scenes with a large amount of downloaded data, in order to avoid the poor download experience of the user due to the client's long waiting period ,
  • the download request of the client is usually processed by asynchronous download.
  • the asynchronous downloading service logic will crazily occupy the memory resources of the JVM (Java Virtual Machine), causing frequent FULL GC (garbage Collection, garbage collection or recycling of useless memory space), which will have a great impact on system performance, and may even affect the normal operation of other functions of the system.
  • JVM Java Virtual Machine
  • FULL GC Garbage Collection, garbage collection or recycling of useless memory space
  • the present invention provides an asynchronous processing method, device, system, computer equipment and storage medium for downloading tasks.
  • the embodiment of the present invention can improve the downloading of the system in scenarios where downloading services are concentrated or the amount of downloading data is large. Performance, reducing the overall performance of the system in the above scenarios.
  • the present invention provides a download task asynchronous processing method according to the first aspect.
  • the method includes: upon receiving a download request from any client, generating a download task corresponding to the download request;
  • the download task is stored in the asynchronous task table; when the preset timing task is triggered, the preset task processing configuration information is determined; multiple target download tasks matching the preset task processing configuration information are determined from the asynchronous task table; each target download task is obtained Corresponding target download data; store the target download data corresponding to each target download task in the download data management device; send the storage address of the target download data corresponding to each target download task in the download data management device to the corresponding target download task The target client, so that the target client corresponding to each target download task downloads the target download data corresponding to each target download task from the download data management device through the storage address received by the target client.
  • the preset task processing configuration information includes a single task processing number Z and task screening information, where Z is a positive integer greater than 0; multiple targets matching the preset task processing configuration information are determined from the asynchronous task table Download tasks include: screening multiple candidate unprocessed download tasks from the asynchronous task table according to task screening information; determining Z candidate unprocessed download tasks from multiple candidate unprocessed download tasks as multiple target download tasks;
  • determining N candidate unprocessed download tasks from a plurality of candidate unprocessed download tasks as multiple target download tasks includes: determining the processing priority corresponding to each candidate unprocessed download task; corresponding to each candidate unprocessed download task Sort the multiple candidate unprocessed download tasks with the processing priority of, and determine the first Z candidate unprocessed download tasks with the highest processing priority among multiple candidate unprocessed download tasks as multiple target download tasks.
  • obtaining target download data corresponding to each target download task includes: obtaining task data corresponding to each target download task; generating query information corresponding to each target download task according to the task data corresponding to each target download task; The query information corresponding to the target download task obtains the target download data corresponding to each target download task.
  • the task data corresponding to each target download task is json string data
  • storing the generated download task in the asynchronous task table includes: storing the task data of the generated download task in the asynchronous task table in the form of a json string Task table; generate query information corresponding to each target download task according to the task data corresponding to each target download task, including: parse the json string data corresponding to each target download task to obtain the analysis result corresponding to each target download task; download for each target The analysis results corresponding to the tasks are assembled and processed, and the query information corresponding to each target download task is obtained.
  • generating a download task corresponding to the download request includes: when a download request from any client is received, determining the download task identifier and user corresponding to the download request ID; query the asynchronous task table; if the asynchronous task table does not include unprocessed download tasks corresponding to the download task ID and user ID, generate the download task corresponding to the download request; if the asynchronous task table includes the download task ID and user ID The corresponding unprocessed download task returns a repeated download prompt message to any client.
  • the download task asynchronous processing method further includes the following steps: responding to the task operation request from the administrator client, determining the download task to be operated and task operation information corresponding to the task operation request; modifying the download to be operated according to the task operation information The task status information of the task.
  • the present invention provides a download task asynchronous processing device.
  • the device includes: a download task generating module for generating a download request corresponding to the download request when receiving a download request from any client.
  • the download task, the generated download task is stored in the asynchronous task table;
  • the target task determination module is used to determine the preset task processing configuration information when the preset timing task is triggered, and determine the preset task processing configuration from the asynchronous task table Multiple target download tasks with matching information; target data acquisition module for acquiring target download data corresponding to each target download task; target data storage module for storing target download data corresponding to each target download task in download data management Module; address sending module, used to send the storage address of the target download data corresponding to each target download task in the download data management module to the target client corresponding to each target download task, so that the target client corresponding to each target download task Download the target download data corresponding to each target download task from the download data management module through the received storage address.
  • the present invention provides a download task asynchronous processing system.
  • the system includes a download request processing module, a download task processing module, and a download data management module;
  • the download task processing module includes one or more Task processing node, any task processing node is configured with timing tasks;
  • download request processing module when receiving a download request from any client, generate a download task corresponding to the download request, and store the generated download task in asynchronous Task table; also used to determine the processed download tasks in the asynchronous task table, and send the storage address corresponding to the processed download task to the client corresponding to the processed download task;
  • any task processing node is used to process the task at this task processing node
  • determine the preset task processing configuration information corresponding to the task processing node determine multiple target download tasks matching the preset task processing configuration information from the asynchronous task table, and obtain the target corresponding to each target download task Download data, store the target download data corresponding to each target download task in the download data management module, and update the storage address of
  • the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor.
  • the processor implements the steps of any of the above-mentioned method embodiments when the computer program is executed. .
  • the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned method embodiments are implemented.
  • the processing system when receiving a download request from any client, the processing system generates a download task corresponding to the download request, and stores the generated download task in the asynchronous task table, so as to slow down the processing system from receiving a large number of The impact of a centralized download request; subsequently, when the preset timed task is triggered, determine the preset task processing configuration information and determine from the asynchronous task table multiple target download tasks that match the preset task processing configuration information, and then be able to Obtain the target download data corresponding to each target download task, and the obtained target download data corresponding to each target download task needs to be stored in the download data management device, and the storage address of the target download data corresponding to each target download task in the download data management device Sent to the target client corresponding to each target download task, so that the target client corresponding to each target download task downloads the target download data corresponding to each target download task from the download data management device through the received storage address, thereby achieving Improve the download performance of the processing system in scenarios where the download business is concentrated
  • FIG. 1 is an application environment diagram of a method for asynchronous processing of download tasks in an embodiment
  • FIG. 2 is a schematic diagram of a flow when the processing system generates a download task in an embodiment
  • FIG. 3 is a structural block diagram of a device for asynchronous processing of download tasks in an embodiment
  • Figure 4 is a structural block diagram of a download task asynchronous processing system in an embodiment
  • Fig. 5 is a diagram of the internal structure of a computer device in an embodiment.
  • the method can be executed by the download task asynchronous processing system (hereinafter referred to as the processing system).
  • the processing system can be composed of a single or multiple Server to achieve.
  • the method includes the following steps:
  • Step S110 When a download request from any client is received, a download task corresponding to the download request is generated, and the generated download task is stored in the asynchronous task table.
  • the user terminal can be a portable computing device such as a smart phone, a tablet computer, or a notebook computer. It can be a computing device such as a desktop computer.
  • the processing system When the processing system receives a download request sent by any client, it will not directly respond to the download request and feed back relevant download data to the client corresponding to the download request, but only needs to use a small amount of hardware resources (such as CPU resources, Memory resources, etc.) to generate the download task corresponding to the download request, and store the generated download task in the asynchronous task table, and then the task processing node included in the processing system for processing the download task will be retrieved in batches from the asynchronous task table The download task is processed, which can slow down the impact on system performance when the processing system receives a large number of concentrated download requests.
  • hardware resources such as CPU resources, Memory resources, etc.
  • the above-mentioned task processing node can be implemented by a single server or multiple servers.
  • the aforementioned asynchronous task table records related information corresponding to multiple download tasks.
  • the asynchronous task table records all download tasks generated by the processing system.
  • the relevant information corresponding to the download task includes at least one of the download task identification, user identification, client information, processing status, processing priority, download task generation time, and download task completion time corresponding to the download task.
  • the processing status corresponding to the download task is used to characterize the processing status of the download task by the task processing node in the processing system, and may include status such as unprocessed, downloading, processed, and abnormal downloading.
  • the processing priority includes multiple pre-divided priorities, such as first/second/third, or low/intermediate/high, etc. The number of priority levels can be set according to actual conditions.
  • Step S111 When a download request from any client is received, Determine the download task ID and user ID corresponding to the download request; Step S112: Query the asynchronous task table; Step S113: If the asynchronous task table does not include unprocessed download tasks corresponding to the download task ID and user ID, generate a corresponding download request Step S114: If the asynchronous task table includes unprocessed download tasks corresponding to the download task ID and the user ID, return a repeated download prompt message to any client.
  • the processing system in this embodiment receives a download request from any client, it first determines the download task ID and user ID corresponding to the received download request, and then queries whether there is a download request in the asynchronous task table. If the unprocessed download task corresponding to the task ID and the user ID (that is, the download task whose processing status is unprocessed) does not exist, it means that the download request received this time is not a request repeatedly submitted by the user, and the processing system can perform subsequent operations.
  • the download task corresponding to the download request is generated; on the contrary, if it exists, it means that the download request received this time is a request repeatedly submitted by the user.
  • the processing system does not need to generate the corresponding download task again, and can submit the download request to The client returns a repeated download prompt message, so that the user does not repeatedly submit download requests.
  • Step S120 When the preset timing task is triggered, the preset task processing configuration information is determined, and multiple target download tasks matching the preset task processing configuration information are determined from the asynchronous task table.
  • the task processing node included in the processing system is pre-configured with timing tasks.
  • the processing system includes multiple task processing nodes, and each task processing node is configured with independent timing tasks.
  • the task execution logic of each timing task is the same, that is, multiple task processing nodes are determined from the asynchronous task table at regular intervals.
  • the download task to be processed (that is, the target download task), and then the determined download task is processed.
  • different timing tasks correspond to different preset task processing configuration information, and each task processing node can determine the download task that it is responsible for processing from the asynchronous task table through the preset task processing configuration information.
  • the unprocessed download tasks in the asynchronous task table can be grouped, and the number of groups is the same as the number of task processing nodes.
  • each task processes The node is responsible for processing unprocessed download tasks of a group, and all download tasks in a group are the download tasks to be processed by the task processing node corresponding to the group after this timed task is triggered (ie, target download tasks).
  • the preset task processing configuration information includes task screening information, and each task processing node determines the download task that it is responsible for processing from the asynchronous task table according to the task screening information included in its corresponding preset task processing configuration information.
  • determining multiple target download tasks matching preset task processing configuration information from the asynchronous task table includes: screening multiple candidate unprocessed download tasks from the asynchronous task table according to task screening information; Among the three candidate unprocessed download tasks, Z candidate unprocessed download tasks are determined as multiple target download tasks;
  • the threshold of the number of single task processing Z can be further set to control the number of download tasks that the task processing node is responsible for processing each time, that is, the preset task processing configuration information includes the number of single task processing and task filtering information.
  • the number of sub-tasks to be processed is a threshold, which can be referred to as Z.
  • Z is a positive integer greater than 0. The specific value of Z can be set and dynamically adjusted by the administrator according to the actual scene.
  • the single task processing quantity of each task processing node needs to be independently configured, and the single task processing quantity of different task processing nodes may be the same or different.
  • the task screening information may include a modulus threshold and a remainder threshold.
  • the modulus threshold is the same as the number of task processing nodes included in the processing system, and the modulus threshold in the task screening information corresponding to each task processing node is the same but the remainder threshold is different.
  • the processing system has three task processing nodes, node A, node B, and node C.
  • the task screening information of node A is (3, 0), that is, the modulus threshold is 3, the remainder threshold is 0, and the task screening information of node B Is (3,1), and the task screening information of node C is (3,2).
  • Each task processing node will traverse each unprocessed download task recorded in the asynchronous task table, and the download task identifier traversed to each unprocessed download task will be modulated based on the modulus threshold. If the remainder threshold is the same, it is determined that the unprocessed download task is a download task that is processed by itself.
  • each task processing node selects multiple candidate unprocessed download tasks from the asynchronous task table through the above operations, it selects Z candidate unprocessed download tasks from the selected candidate unprocessed download tasks as the secondary process to be processed. Download tasks (ie, multiple target download tasks).
  • the task processing node when it filters download tasks, it can sort the download task generation time corresponding to each candidate unprocessed download task, and then sort the top Z with the largest time difference between the download task generation time and the current system time
  • the candidate unprocessed download task is determined as the download task to be processed this time.
  • the greater the time difference between the download task generation time of the download task and the current system time the longer the user’s waiting time. For this reason, this type of download task can be processed first to avoid letting the user wait too long and affect the user’s Download experience.
  • determining N candidate unprocessed download tasks from multiple candidate unprocessed download tasks as multiple target download tasks includes: determining the processing priority of each candidate unprocessed download task; corresponding to each candidate unprocessed download task Sort the multiple candidate unprocessed download tasks with the processing priority of, and determine the first Z candidate unprocessed download tasks with the highest processing priority among multiple candidate unprocessed download tasks as multiple target download tasks.
  • the task processing node when it filters download tasks, it can sort the download tasks according to the processing priority of each candidate unprocessed download task, and then determine the first Z candidate unprocessed download tasks with the highest processing priority as the download to be processed. Task.
  • the higher the processing priority of the download task the higher the urgency of the download task for the user. For this reason, this type of download task can be processed first.
  • the timing tasks corresponding to each task processing node are independent, and the triggering rules of each timing task may be the same or different.
  • the trigger rule of each timed task can be triggered every 30 seconds, or the trigger rule of some timed tasks is triggered every 30 seconds, and the trigger rule of the rest of the timed tasks is triggered every 20 seconds, etc.
  • the task processing node since this embodiment decouples the response operation of the download request from the processing operation of the download task, the task processing node only needs to be responsible for processing the download task. Therefore, the number of task processing nodes included in the processing system can be easily expanded or expanded. reduce. Specifically, the number of task processing nodes included in the processing system can be dynamically adjusted by the administrator according to the actual situation to meet the actual needs of processing download tasks. For example, when the number of unprocessed download tasks is too large, the administrator can expand the task processing nodes. For example, the original 3 task processing nodes are expanded to 5 task processing nodes. It should be noted that when expanding or reducing the number of task processing nodes, the preset task processing configuration information corresponding to each task processing node needs to be adjusted accordingly to regroup unprocessed download tasks in the asynchronous task table.
  • Step S130 Obtain target download data corresponding to each target download task.
  • the task processing node included in the processing system determines the multiple target download tasks that need to be processed when the timing task is triggered, and then further obtains the target download data corresponding to each target download task.
  • obtaining target download data corresponding to each target download task includes
  • each task processing node may sequentially obtain the target download data of its corresponding multiple target download tasks, that is, after obtaining the target download data corresponding to one target download task, then obtain the target download data corresponding to the next target download task. Target download data until the target download data corresponding to the last target download task is obtained.
  • the task processing node when it obtains the target download data corresponding to the target download task, it first obtains the task data corresponding to the target download task from the asynchronous task table.
  • the task data is task detail information describing the download task. For example, the download task needs to be downloaded. Descriptive information of which data, and then generate query information based on the task data.
  • the query information corresponding to each target download task is generated according to the task data corresponding to each target download task, including: parsing the json string data corresponding to each target download task to obtain the analysis result corresponding to each target download task; downloading to each target
  • the analysis results corresponding to the tasks are assembled and processed, and the query information corresponding to each target download task is obtained.
  • the task data corresponding to the generated download task will be stored in the asynchronous task table in the form of a json (JavaScript Object Notation, JS object notation) string. Therefore, when the task processing node starts to process the download task, the task data corresponding to each target download task obtained from the asynchronous task table is json string data, so the json string data corresponding to the target download task needs to be parsed, and then Assemble the analysis results obtained by the analysis into query information corresponding to the target download task.
  • the query information may be a SQL (Structured Query Language, structured query language) query statement, and the obtained SQL query statement can query the corresponding data from the database (that is, the target download data).
  • Step S140 Store the target download data corresponding to each target download task in the download data management device.
  • the processing system adopts the asynchronous download mode. After the user submits the download request, the connection with the processing system can be disconnected, and there is no need to keep the connection to wait for the download result of the download task corresponding to the download request. After the processing system obtains the target download data corresponding to the target download task through the task processing node, the obtained target download data is stored in the download data management device.
  • the download data management device may be a file server.
  • Step S150 Send the storage address of the target download data corresponding to each target download task in the download data management device to the target client corresponding to each target download task, so that the target client corresponding to each target download task can receive The storage address downloads the target download data corresponding to each target download task from the download data management device.
  • the processing system stores the target download data corresponding to each target download task in the download data management device, it sends the storage address of the target download data corresponding to each target download task in the download data management device to each target The target client corresponding to the download task.
  • each target client After each target client receives the storage address, each target client establishes a connection with the download data management device, and has downloaded the download data corresponding to the storage address from the download data management device, that is, the target corresponding to each target download task Download the data.
  • the processing system when receiving a download request from any client, the processing system generates a download task corresponding to the download request, and stores the generated download task in the asynchronous task table, so as to slow down the processing system from receiving a large number of concentrated tasks.
  • the impact of the download request subsequently, when the preset timed task is triggered, determine the preset task processing configuration information and determine from the asynchronous task table multiple target download tasks that match the preset task processing configuration information, and then get The target download data corresponding to each target download task, the obtained target download data corresponding to each target download task needs to be stored in the download data management device, and the target download data corresponding to each target download task is sent to the storage address in the download data management device Give the target client corresponding to each target download task, so that the target client corresponding to each target download task downloads the target download data corresponding to each target download task from the download data management device through the received storage address, thereby achieving improvement
  • the download performance of the processing system in scenarios where the download business is concentrated or the download data volume is large can reduce the impact of the download business concentration or the large download data volume on the overall performance of the processing system.
  • the download task asynchronous processing method further includes: responding to a task operation request from the administrator client, determining the download task to be operated and task operation information corresponding to the task operation request; modifying the download task to be operated according to the task operation information Task status information.
  • the administrator sends a task operation request to the processing system through his user terminal (ie, the administrator client) to operate one or more download tasks.
  • the processing system determines the download task (that is, the download task to be operated) and the task operation information corresponding to the task operation request that need to perform related operations.
  • the task operation information may include task operation type and task operation content, and the task operation type may include operation types such as adjustment of processing priority and adjustment of processing status.
  • the administrator can adjust the processing priority of a certain download task from the lowest priority to the highest priority, so that the download task can be processed by the related task processing node more quickly.
  • the administrator can cancel or suspend the processing of a download task that has multiple download exceptions.
  • the processing system can monitor the number of unprocessed download tasks in the asynchronous task table, and if the number of unprocessed download tasks exceeds a preset threshold, it will send a task backlog reminder to the administrator client to make management
  • the operator handles the task backlog, for example, performing the above operations on certain download tasks, or expanding the number of task processing nodes in the processing system, or increasing the number of single task processing Z of task processing nodes.
  • a download task asynchronous processing device which includes the following modules:
  • the download task generation module 110 is used to generate a download task corresponding to the download request when receiving a download request from any client, and store the generated download task in the asynchronous task table;
  • the target task determination module 120 is used to When the task is triggered during setting, the preset task processing configuration information is determined, and multiple target download tasks matching the preset task processing configuration information are determined from the asynchronous task table;
  • the target data acquisition module 130 is used to obtain the corresponding target download task Target download data;
  • target data storage module 140 used to store the target download data corresponding to each target download task in the download data management module; address sending module 150, used to store the target download data corresponding to each target download task in the download data
  • the storage address in the management module is sent to the target client corresponding to each target download task, so that the target client corresponding to each target download task downloads the target corresponding to each target download task from the download data management module through the received storage address Download the data.
  • the preset task processing configuration information includes a single task processing quantity Z and task screening information, where Z is a positive integer greater than zero.
  • the target task determination module includes: a candidate task screening sub-module, which is used to filter out multiple candidate unprocessed download tasks from the asynchronous task table according to task screening information; the target task determination sub-module is used to download multiple candidate unprocessed download tasks Determine Z candidate unprocessed download tasks as multiple target download tasks.
  • the target task determination sub-module includes: a priority determining unit for determining the processing priority corresponding to each candidate unprocessed download task; and a sorting unit for determining the processing priority corresponding to each candidate unprocessed download task Ranks the multiple candidate unprocessed download tasks; the target task determining unit is used to determine the first Z candidate unprocessed download tasks with the highest processing priority among multiple candidate unprocessed download tasks as multiple target download tasks.
  • the target data acquisition module includes: a task data acquisition sub-module for acquiring task data corresponding to each target download task; a query information generation sub-module for generating each task data corresponding to each target download task The query information corresponding to the target download task; the target data obtaining submodule is used to obtain the target download data corresponding to each target download task according to the query information corresponding to each target download task.
  • the task data corresponding to each target download task is json string data.
  • the download task generation module is used to store the task data of the generated download task in the asynchronous task table in the form of a json string.
  • the query information generation sub-module includes: a parsing unit, which is used to parse the json string data corresponding to each target download task, to obtain the analysis result corresponding to each target download task; the query information generation unit, which is used to analyze each target download task The result is assembled and processed, and query information corresponding to each target download task is obtained.
  • the download task generation module includes: an identification determination sub-module, which is used to determine the download task identification and user identification corresponding to the download request when a download request from any client is received; and the query sub-module is used to Query the asynchronous task table; the download task generation sub-module is used to generate the download task corresponding to the download request when the asynchronous task table does not include the unprocessed download task corresponding to the download task ID and the user ID; the prompt message sending sub-module, When the asynchronous task table includes unprocessed download tasks corresponding to the download task identifier and the user identifier, it is used to return a repeated download prompt message to any client.
  • an identification determination sub-module which is used to determine the download task identification and user identification corresponding to the download request when a download request from any client is received
  • the query sub-module is used to Query the asynchronous task table
  • the download task generation sub-module is used to generate the download task corresponding to the download request when the asynchronous task table does
  • the download task asynchronous processing device further includes: an operation request response module, which is used to respond to a task operation request from the administrator client and determine the download task to be operated and task operation information corresponding to the task operation request; and a task state modification module , Used to modify the task status information of the download task to be operated according to the task operation information.
  • each module in the above-mentioned download task asynchronous processing device can be implemented in whole or in part by software, hardware, and a combination thereof.
  • the foregoing modules may be embedded in the form of hardware or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.
  • a download task asynchronous processing system includes a download request processing module 210, a download task processing module 220, and a download data management module 230; the download task processing module includes a Or multiple task processing nodes, any task processing node is configured with a timing task; the download request processing module 210 is used to generate a download task corresponding to the download request when receiving a download request from any client, and download the generated download The task is stored in the asynchronous task table; it is also used to determine the processed download task in the asynchronous task table, and the storage address corresponding to the processed download task is sent to the client corresponding to the processed download task; any task processing node is used in When the timing task of the task processing node is triggered, determine the preset task processing configuration information corresponding to the task processing node, determine multiple target download tasks matching the preset task processing configuration information from the asynchronous task table, and obtain each target download Target download data corresponding to the task, store the target download data corresponding to each target
  • the preset task processing configuration information includes a single task processing number Z and task screening information, Z is a positive integer greater than 0; any task processing node is determined from the asynchronous task table and the preset task processing When configuring the operation of multiple target download tasks matching the information, it is specifically used to: filter multiple candidate unprocessed download tasks from the asynchronous task table according to the task screening information; determine Z candidate unprocessed download tasks from the multiple candidate unprocessed download tasks Handle download tasks as multiple target download tasks.
  • any task processing node when any task processing node performs an operation of determining N candidate unprocessed download tasks from multiple candidate unprocessed download tasks as multiple target download tasks, it is specifically used to: determine each candidate unprocessed download The processing priority corresponding to the task; sort multiple candidate unprocessed download tasks according to the processing priority corresponding to each candidate unprocessed download task; among multiple candidate unprocessed download tasks, the first Z candidate unprocessed download tasks with the highest processing priority The processing download task is determined to be multiple target download tasks.
  • any task processing node when any task processing node performs the operation of obtaining target download data corresponding to each target download task, it is specifically used to: obtain task data corresponding to each target download task; download task data corresponding to each target download task Generate query information corresponding to each target download task; obtain target download data corresponding to each target download task according to the query information corresponding to each target download task; in one embodiment, the task data corresponding to each target download task is json string data ;
  • the download request processing module executes the operation of storing the generated download task in the asynchronous task table, it is specifically used to: store the task data of the generated download task in the asynchronous task table in the form of a json string; any task processing node
  • the analysis results corresponding to each target download task are assembled and processed to
  • the download request processing module executes the operation of generating a download task corresponding to the download request when receiving a download request from any client, it is specifically used for: when receiving a download request from any client , Determine the download task ID and user ID corresponding to the download request; query the asynchronous task table; if the asynchronous task table does not include unprocessed download tasks corresponding to the download task ID and user ID, generate the download task corresponding to the download request
  • a repeated download prompt message is returned to any client.
  • the download task asynchronous processing system further includes a download task management module 240.
  • the download task management module 240 is used to respond to a task operation request from the administrator client, determine the download task to be operated and task operation information corresponding to the task operation request, and modify the task status information of the download task to be operated according to the task operation information.
  • download task asynchronous processing system For the specific limitation of the download task asynchronous processing system, please refer to the above limitation on the download task asynchronous processing method, which will not be repeated here.
  • the various modules in the above-mentioned download task asynchronous processing system can be implemented in whole or in part by software, hardware and a combination thereof.
  • the foregoing modules may be embedded in the form of hardware or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.
  • a computer device in one embodiment, is provided, and its internal structure diagram may be as shown in FIG. 5.
  • the computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus.
  • the processor of the computer device is used to provide calculation and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, a computer program, and a database.
  • the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the database of the computer equipment is used for data such as asynchronous task tables.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an asynchronous processing method of the download task is realized.
  • FIG. 5 is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied.
  • the specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.
  • a computer device including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the following steps when the processor executes the computer program:
  • a download task corresponding to the download request is generated; the generated download task is stored in the asynchronous task table; when the preset timing task is triggered, the preset task processing configuration information is determined; Identify multiple target download tasks matching the preset task processing configuration information in the task table; obtain the target download data corresponding to each target download task; store the target download data corresponding to each target download task in the download data management device; store each target The storage address of the target download data corresponding to the download task in the download data management device is sent to the target client corresponding to each target download task, so that the target client corresponding to each target download task can manage the download data from the storage address it receives The device downloads target download data corresponding to each target download task.
  • the preset task processing configuration information includes a single task processing number Z and task screening information, where Z is a positive integer greater than 0; the processor executes a computer program to realize the determination and preset task processing from the asynchronous task table
  • the following steps are also implemented: filter multiple candidate unprocessed download tasks from the asynchronous task table according to task screening information; determine Z candidate unprocessed download tasks from multiple candidate unprocessed download tasks The download task serves as a multiple target download task.
  • the processor executes a computer program to determine N candidate unprocessed download tasks from a plurality of candidate unprocessed download tasks as multiple target download tasks
  • the following steps are also implemented: determine each candidate unprocessed download task Corresponding processing priority; sort multiple candidate unprocessed download tasks according to the processing priority corresponding to each candidate unprocessed download task; among multiple candidate unprocessed download tasks, the first Z candidates with the highest processing priority are unprocessed
  • the download task is determined as multiple target download tasks.
  • the processor when the processor executes the computer program to obtain the target download data corresponding to each target download task, it also implements the following steps: obtain the task data corresponding to each target download task; generate according to the task data corresponding to each target download task Query information corresponding to each target download task; obtain target download data corresponding to each target download task according to the query information corresponding to each target download task.
  • the task data corresponding to each target download task is json string data; when the processor executes the computer program to store the generated download task in the asynchronous task table, it also implements the following steps:
  • the task data is stored in the asynchronous task table in the form of a json string;
  • the processor executes the computer program to generate query information corresponding to each target download task according to the task data corresponding to each target download task, and the following steps are also implemented: Parse each target download
  • the json string data corresponding to the task obtains the analysis result corresponding to each target download task; the analysis results corresponding to each target download task are assembled and processed to obtain the query information corresponding to each target download task.
  • the processor executes the computer program to realize that when a download task corresponding to the download request is generated when a download request from any client is received, the following steps are also implemented: when a download request from any client is received , Determine the download task ID and user ID corresponding to the download request; query the asynchronous task table; if the asynchronous task table does not include unprocessed download tasks corresponding to the download task ID and user ID, generate the download task corresponding to the download request; if asynchronous The task table includes unprocessed download tasks corresponding to the download task identifier and the user identifier, and a repeated download prompt message is returned to any client.
  • the processor when the processor executes the computer program, the processor further implements the following steps: responding to the task operation request from the administrator client, determining the download task and task operation information corresponding to the task operation request; modifying the task operation information according to the task operation information Download task status information of the task.
  • a computer-readable storage medium on which a computer program is stored.
  • the following steps are implemented: upon receiving a download request from any client, generating and downloading request Corresponding download tasks; store the generated download tasks in the asynchronous task table; determine the preset task processing configuration information when the preset timing task is triggered; determine multiple targets matching the preset task processing configuration information from the asynchronous task table Download task; obtain the target download data corresponding to each target download task; store the target download data corresponding to each target download task in the download data management device; store the storage address of the target download data corresponding to each target download task in the download data management device It is sent to the target client corresponding to each target download task, so that the target client corresponding to each target download task downloads the target download data corresponding to each target download task from the download data management device through the received storage address.
  • the preset task processing configuration information includes a single task processing number Z and task screening information, Z is a positive integer greater than 0; the computer program is executed by the processor, and the preset task processing is determined from the asynchronous task table.
  • the following steps are also implemented: filter multiple candidate unprocessed download tasks from the asynchronous task table according to task screening information; determine Z candidate unprocessed download tasks from multiple candidate unprocessed download tasks The download task serves as a multiple target download task.
  • the computer program is executed by the processor, and when N candidate unprocessed download tasks are determined as multiple target download tasks from the multiple candidate unprocessed download tasks, the following steps are also implemented: determine each candidate unprocessed download task Corresponding processing priority; sort multiple candidate unprocessed download tasks according to the processing priority corresponding to each candidate unprocessed download task; among multiple candidate unprocessed download tasks, the first Z candidates with the highest processing priority are unprocessed
  • the download task is determined as multiple target download tasks.
  • the following steps are also implemented: obtain the task data corresponding to each target download task; generate according to the task data corresponding to each target download task Query information corresponding to each target download task; obtain target download data corresponding to each target download task according to the query information corresponding to each target download task.
  • the task data corresponding to each target download task is json string data; when the computer program is executed by the processor and the generated download task is stored in the asynchronous task table, the following steps are also implemented:
  • the task data is stored in the asynchronous task table in the form of a json string; when the processor executes the computer program and generates the query information corresponding to each target download task according to the task data corresponding to each target download task, the following steps are also implemented: Parse each target download task Corresponding json string data, obtain the analysis results corresponding to each target download task; perform assembly processing on the analysis results corresponding to each target download task, and obtain query information corresponding to each target download task.
  • the computer program is executed by the processor, and when a download task corresponding to the download request is generated when a download request from any client is received, the following steps are further implemented: when a download request from any client is received , Determine the download task ID and user ID corresponding to the download request; query the asynchronous task table; if the asynchronous task table does not include unprocessed download tasks corresponding to the download task ID and user ID, generate the download task corresponding to the download request; if asynchronous The task table includes unprocessed download tasks corresponding to the download task identifier and the user identifier, and a repeated download prompt message is returned to any client.
  • the following steps are further implemented: in response to the task operation request from the administrator client, determine the download task to be operated and task operation information corresponding to the task operation request; modify the task operation information according to the task operation information.
  • the task status information of the operation download task in response to the task operation request from the administrator client, determine the download task to be operated and task operation information corresponding to the task operation request; modify the task operation information according to the task operation information.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本申请具体涉及一种下载任务异步处理方法、装置、系统和存储介质。所述方法包括：在接收到来自任一客户端的下载请求时，生成与该请求对应的下载任务，将生成的下载任务存入异步任务表；在预设定时任务触发时，确定预设任务处理配置信息并从该表中确定与该理配置信息匹配的多个目标下载任务；获取各个目标下载任务对应的目标下载数据；将各个目标下载任务对应的目标下载数据存入下载数据管理设备；将各个目标下载任务对应的目标下载数据在该管理设备中的存储地址发送给各个目标下载任务对应的目标客户端。本发明可提升系统在下载业务集中或下载数据量大的场景下的下载性能，降低系统整体性能在上述场景中受到的影响。

Description

下载任务异步处理方法、装置、系统和存储介质 技术领域
本申请涉及软件开发领域，特别是涉及一种下载任务异步处理方法、装置、系统和存储介质。
 背景技术
在开发业务系统功能时无可避免地会涉及到下载功能。通常，对于下载数据量较小的场景，会采用同步下载的方式来处理客户端的下载请求；对于下载数据量很大的场景, 为了避免因客户端陷入长时间的等待而导致用户下载体验不佳，则通常采用异步下载的方式来处理客户端的下载请求。
 技术问题
对于下载业务集中或下载数据量大的场景，即使对大量的下载业务进行分批处理，异步下载业务逻辑也会对JVM（Java Virtual Machine，Java虚拟机）的内存资源疯狂占用，进而造成频繁的FULL GC（garbage collection，垃圾回收或回收无用内存空间），这会对系统性能造成很大影响,甚至还有可能影响系统其他功能的正常运行。
 技术解决方案
本发明针对现有技术的缺点，提供了一种下载任务异步处理方法、装置、系统、计算机设备和存储介质，本发明实施例可提升系统在下载业务集中或下载数据量大的场景下的下载性能，降低系统整体性能在上述场景中受到的影响。
本发明根据第一方面提供了一种下载任务异步处理方法，在一个实施例中，该方法包括：在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务；将生成的下载任务存入异步任务表；在预设定时任务触发时，确定预设任务处理配置信息；从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务；获取各个目标下载任务对应的目标下载数据；将各个目标下载任务对应的目标下载数据存入下载数据管理设备；将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据。
在一个实施例中，预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，包括：根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务；
优选地，从多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务，包括：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
在一个实施例中，获取各个目标下载任务对应的目标下载数据，包括：获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
在一个实施例中，各个目标下载任务对应的任务数据是json字符串数据；将生成的下载任务存入异步任务表，包括：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息，包括：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
在一个实施例中，在接收到来自任一客户端的下载请求时生成与下载请求对应的下载任务，包括：在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；查询异步任务表；若异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务，生成与下载请求对应的下载任务；若异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务，向任一客户端返回重复下载提示消息。
在一个实施例中，该下载任务异步处理方法还包括以下步骤：响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息；根据任务操作信息修改待操作下载任务的任务状态信息。
本发明根据第二方面提供了一种下载任务异步处理装置，在一个实施例中，该装置包括：下载任务生成模块，用于在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表；目标任务确定模块，用于在预设定时任务触发时，确定预设任务处理配置信息，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务；目标数据获取模块，用于获取各个目标下载任务对应的目标下载数据；目标数据存入模块，用于将各个目标下载任务对应的目标下载数据存入下载数据管理模块；地址发送模块，用于将各个目标下载任务对应的目标下载数据在下载数据管理模块中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理模块下载与各个目标下载任务对应的目标下载数据。
本发明根据第三方面提供了一种下载任务异步处理系统，在一个实施例中，该系统包括下载请求处理模块、下载任务处理模块、以及下载数据管理模块；下载任务处理模块包括一个或多个任务处理节点，任一个任务处理节点配置有定时任务；下载请求处理模块，用于在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表；还用于确定异步任务表中的已处理下载任务，将已处理下载任务对应的存储地址发送给已处理下载任务对应的客户端；任一个任务处理节点，用于在本任务处理节点的定时任务触发时，确定与本任务处理节点对应的预设任务处理配置信息，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，获取各个目标下载任务对应的目标下载数据，将各个目标下载任务对应的目标下载数据存入下载数据管理模块，将各个目标下载任务对应的目标下载数据在下载数据管理模块中的存储地址更新到异步任务表中；下载数据管理模块，用于存储下载数据；还用于在接收到来自任一客户端的存储地址时，将与存储地址对应的下载数据发送给该任一客户端。
本发明根据第四方面提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述任一方法的实施例的步骤。
本发明根据第五方面提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现上述任一方法的实施例的步骤。
 有益效果
在本发明实施例中，处理系统在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表，以减缓处理系统在接收到大量且集中的下载请求时所受到的冲击；后续在预设定时任务触发时，确定预设任务处理配置信息并从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，进而能够获得各个目标下载任务对应的目标下载数据，获得的各个目标下载任务对应的目标下载数据需要存入下载数据管理设备，并将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据，从而能够实现提高下载业务集中或下载数据量大的场景下处理系统的下载性能，降低下载业务集中或下载数据量大等情况对处理系统整体性能的影响。
 附图说明
图1为一个实施例中一种下载任务异步处理方法的应用环境图；
图2为一个实施例中处理系统生成下载任务时的流程示意图；
图3为一个实施例中一种下载任务异步处理装置的结构框图；
图4为一个实施例中一种下载任务异步处理系统的结构框图；
图5为一个实施例中计算机设备的内部结构图。
 本发明的实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅用以解释本申请，并不用于限定本申请。
如图1所示，是本发明一个实施例提供的一种下载任务异步处理方法，该方法可以通过下载任务异步处理系统（以下简称为处理系统）来执行，处理系统可以由单台或多台服务器来实现。该方法包括以下步骤：
步骤S110：在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表。在本实施例中，当用户需要下载相关数据时，可以通过其用户终端（即客户端）向处理系统发送下载请求，用户终端可以是智能手机、平板电脑、笔记本电脑等便携性计算设备，也可以是台式电脑等计算设备。
处理系统在收到任一客户端发送的下载请求时，不会直接响应该下载请求并向该下载请求对应的客户端反馈相关下载数据，而只需要使用少量的硬件资源（比如，CPU资源、内存资源等）来生成与该下载请求对应的下载任务，并将生成的下载任务存入异步任务表，后续再由处理系统包括的用于处理下载任务的任务处理节点从异步任务表中批量捞取下载任务进行处理，这样能够在处理系统接收到大量且集中的下载请求时减缓其系统性能受到的冲击。
上述的任务处理节点可以由单台服务器或多台服务器来实现。上述的异步任务表记录有多个下载任务对应的相关信息。
在一个实施方式中，异步任务表记录处理系统生成的所有下载任务。下载任务对应的相关信息包括下载任务对应的下载任务标识、用户标识、客户端信息、处理状态、处理优先级、下载任务生成时间以及下载任务完成时间中的至少一项信息。具体地，下载任务对应的处理状态用于表征处理系统中的任务处理节点对下载任务的处理状态，可以包括未处理、下载中、已处理和下载异常等状态。处理优先级包括预先划分好的多个不同等级的优先级，比如，一级/二级/三级，又比如，低级/中级/高级等，优先级的等级数量可以根据实际情况进行设置。
在一个实施方式中，如图2所示，在接收到来自任一客户端的下载请求时生成与下载请求对应的下载任务，包括以下步骤：步骤S111：在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；步骤S112：查询异步任务表；步骤S113：若异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务，生成与下载请求对应的下载任务；步骤S114：若异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务，向该任一客户端返回重复下载提示消息。
考虑到用户在等待下载结果的过程中，有可能会多次向处理系统提交下载请求，这样会使得处理系统生成多个相同的下载任务，这会造成系统资源的浪费。为此，本实施方式中处理系统在接收到来自任一客户端的下载请求时，先确定接收到的该下载请求所对应的下载任务标识和用户标识，然后查询异步任务表中是否存在与该下载任务标识和用户标识对应的未处理下载任务（即处理状态为未处理的下载任务），如果没有存在，那么表明本次接收到的下载请求不是用户重复提交的请求，处理系统可以执行后续操作，即生成与该下载请求对应的下载任务；反之，如果有存在，那么表明本次接收到的下载请求是用户重复提交的请求，处理系统不需要再生成相应的下载任务，可以向提交该下载请求的客户端返回重复下载提示消息，以使用户不再重复提交下载请求。
步骤S120：在预设定时任务触发时，确定预设任务处理配置信息，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务。在本实施例中，处理系统包括的任务处理节点预先配置有定时任务。
在一个实施方式中，处理系统包括多个任务处理节点，任一个任务处理节点配置有独立的定时任务，各个定时任务的任务执行逻辑相同，即每隔一段时间就从异步任务表中确定多个要处理的下载任务（即目标下载任务），然后对确定出的下载任务进行处理。具体地，不同定时任务对应有不同的预设任务处理配置信息，各个任务处理节点通过预设任务处理配置信息可以从异步任务表中确定自身负责处理的下载任务。通过为各个任务处理节点配置不同的预设任务处理配置信息，可以实现将异步任务表中未处理下载任务进行分组，分组数量与任务处理节点的数量一致，即本实施方式中，每个任务处理节点负责处理一个组的未处理下载任务，一个分组中的所有下载任务都是该分组对应的任务处理节点在本次定时任务触发后要处理的下载任务（即目标下载任务）。
具体地，预设任务处理配置信息包括任务筛选信息，各个任务处理节点会根据自己对应的预设任务处理配置信息所包括任务筛选信息从异步任务表中确定出自己负责处理的下载任务。
在一个实施方式中，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，包括：根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务；
在本实施方式中，考虑到实际场景中一个分组包括的下载任务的数量有可能非常庞大，任务处理节点无法一次将组内的所有下载任务都处理完成。为此，可以进一步设置单次任务处理数量Z这一阈值，以控制任务处理节点每次负责处理的下载任务的数量，即预设任务处理配置信息包括单次任务处理数量和任务筛选信息，单次任务处理数量是一个阈值，可以以Z代称，Z为大于0的正整数，Z的具体取值可以由管理员根据实际场景设置以及动态调整。进一步地，各个任务处理节点的单次任务处理数量需要独立配置，不同任务处理节点的单次任务处理数量可以是相同的，也可以是不相同。
其中，任务筛选信息可以包括取模阈值和余数阈值。该取模阈值与处理系统包括的任务处理节点的数量相同，各个任务处理节点对应的任务筛选信息中的取模阈值相同而余数阈值不同。比如，处理系统有三个任务处理节点，节点A、节点B和节点C，节点A 的任务筛选信息为（3,0），即取模阈值为3，余数阈值为0，节点B的任务筛选信息为（3,1），节点C的任务筛选信息为（3,2）。各任务处理节点会遍历异步任务表记录的每个未处理下载任务，将遍历到每个未处理下载任务的下载任务标识基于该取模阈值进行取模，如果取模后得到的余数与自身的余数阈值相同，则确定该未处理下载任务是自己负责处理的下载任务。
进一步，各任务处理节点通过上述操作从异步任务表中筛选出多个候选未处理下载任务之后，从筛选出来的候选未处理下载任务中筛选出Z个候选未处理下载任务作为本次要处理的下载任务（即多个目标下载任务）。
可能地，任务处理节点筛选下载任务的时候，可以按照各个候选未处理下载任务对应的下载任务生成时间进行排序，然后将下载任务生成时间与当前系统时间之间的时间差值最大的前Z个候选未处理下载任务确定为本次要处理的下载任务。下载任务的下载任务生成时间与当前系统时间之间的时间差值越大，说明用户的等待时间越长，为此可以优先对这类下载任务进行处理，避免让用户等待太久而影响用户的下载体验。
可能地，从多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务，包括：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
其中，任务处理节点筛选下载任务的时候，可以按照各个候选未处理下载任务对应的处理优先级进行排序，然后将处理优先级最高的前Z个候选未处理下载任务确定为本次要处理的下载任务。下载任务的处理优先级越高，该下载任务对于用户来说紧急程度越高，为此可以优先对这类下载任务进行处理。
在一个实施方式中，各个任务处理节点对应的定时任务是独立的，各个定时任务的触发规则可以是相同的也可以是不同的。比如，各个定时任务的触发规则可以都是每隔30秒触发一次，或者部分定时任务的触发规则为每隔30秒触发一次、其余部分定时任务的触发规则为每隔20秒触发一次等。
进一步地，由于本实施例将下载请求的响应操作与下载任务的处理操作进行解耦，任务处理节点只需要负责处理下载任务，因此，处理系统包括的任务处理节点的数量可以方便地进行扩展或缩减。具体地，处理系统包括的任务处理节点的数量可以由管理员根据实际情况动态调整，以满足实际的处理下载任务的需求，比如，在未处理下载任务数量过多时，管理员可以扩展任务处理节点的数量，譬如将原先3个任务处理节点扩展为5个任务处理节点。需要说明的是，在扩展或缩减任务处理节点的数量时，需要相应地对各任务处理节点对应的预设任务处理配置信息进行调整，以将异步任务表中未处理下载任务重新进行分组。
步骤S130：获取各个目标下载任务对应的目标下载数据。在本实施例中，处理系统包括的任务处理节点在确定了本次定时任务触发时需要处理的多个目标下载任务后，进一步获取各个目标下载任务对应的目标下载数据。
在一个实施方式中，获取各个目标下载任务对应的目标下载数据，包括
获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
在本实施方式中，各任务处理节点可以是依次获取其对应的多个目标下载任务的目标下载数据，即获取完一个目标下载任务对应的目标下载数据后，再获取下一个目标下载任务对应的目标下载数据，直至获取完最后一个目标下载任务对应的目标下载数据。
具体地，任务处理节点在获取目标下载任务对应的目标下载数据时，先从异步任务表中获取目标下载任务对应的任务数据，任务数据是描述下载任务的任务详情信息，比如，下载任务需要下载哪些数据的描述信息，之后再根据该任务数据来生成查询信息。
具体地，根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息，包括：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
其中，处理系统在接收到客户端的下载请求后，会将生成的下载任务对应的任务数据以json（JavaScript Object Notation，JS 对象简谱）字符串的形式存入异步任务表。因此，当任务处理节点开始处理下载任务时，从异步任务表中获取到的各个目标下载任务对应的任务数据是json字符串数据，所以需要对目标下载任务对应的json字符串数据进行解析，然后将解析得到的解析结果拼装成目标下载任务对应的查询信息。可能地，查询信息可以是SQL（Structured Query Language，结构化查询语言）查询语句，进而得到的该SQL查询语句可以从数据库中查询到相应的数据（即目标下载数据）。
步骤S140：将各个目标下载任务对应的目标下载数据存入下载数据管理设备。在本实施例中，处理系统采用的是异步下载方式，用户提交下载请求后即可断开与处理系统的连接，无需一直保持连接来等待该下载请求对应的下载任务的下载结果。当处理系统通过任务处理节点获得目标下载任务对应的目标下载数据后，将获得的目标下载数据存入下载数据管理设备中。其中，下载数据管理设备可以是文件服务器。
步骤S150：将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据。在本实施例中，处理系统将各个目标下载任务对应的目标下载数据存到下载数据管理设备后，将各个目标下载任务对应的目标下载数据在该下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端。当各个目标客户端收到存储地址后，各个目标客户端与该下载数据管理设备建立连接，已从该下载数据管理设备中下载该存储地址对应的下载数据，即与各个目标下载任务对应的目标下载数据。
本发明实施例中，处理系统在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表，以减缓处理系统在接收到大量且集中的下载请求时所受到的冲击；后续在预设定时任务触发时，确定预设任务处理配置信息并从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，进而能够获得各个目标下载任务对应的目标下载数据，获得的各个目标下载任务对应的目标下载数据需要存入下载数据管理设备，并将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据，从而能够实现提高下载业务集中或下载数据量大的场景下处理系统的下载性能，降低下载业务集中或下载数据量大等情况对处理系统整体性能的影响。
在一个实施例中，该下载任务异步处理方法还包括：响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息；根据任务操作信息修改待操作下载任务的任务状态信息。
在本实施例中，管理员通过其用户终端（即管理员客户端）向处理系统发送任务操作请求，以对某个或多个下载任务进行操作。
具体地，处理系统在接收到任务操作请求之后，确定与该任务操作请求对应的需要执行相关操作的下载任务（即待操作下载任务）以及任务操作信息。任务操作信息可以包括任务操作类型和任务操作内容，任务操作类型可以包括处理优先级调整、处理状态调整等操作类型。比如，管理员可以将某个下载任务的处理优先级从最低级别的优先级调整为最高级别的优先级，以使该下载任务能够更快地被相关任务处理节点处理。又比如，管理员可以取消或者暂停处理某个多次出现下载异常的下载任务。
进一步地，在一个实施例中，处理系统可以监控异步任务表中未处理下载任务的数量，如果未处理下载任务的数量超过预设阈值，就向管理员客户端发送任务积压提醒，以使管理员对任务积压情况进行处理，比如，对某些下载任务进行上述操作，或者扩展处理系统中的任务处理节点的数量、或者提高任务处理节点的单次任务处理数量Z。
在一个实施例中，如图3所示，提供了一种下载任务异步处理装置，包括以下模块：
下载任务生成模块110，用于在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表；目标任务确定模块120，用于在预设定时任务触发时，确定预设任务处理配置信息，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务；目标数据获取模块130，用于获取各个目标下载任务对应的目标下载数据；目标数据存入模块140，用于将各个目标下载任务对应的目标下载数据存入下载数据管理模块；地址发送模块150，用于将各个目标下载任务对应的目标下载数据在下载数据管理模块中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理模块下载与各个目标下载任务对应的目标下载数据。
在一个实施例中，预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数。目标任务确定模块，包括：候选任务筛选子模块，用于根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；目标任务确定子模块，用于从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务。
在一个实施例中，目标任务确定子模块，包括：优先级确定单元，用于确定各个候选未处理下载任务对应的处理优先级；排序单元，用于按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；目标任务确定单元，用于将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
在一个实施例中，目标数据获取模块，包括：任务数据获取子模块，用于获取各个目标下载任务对应的任务数据；查询信息生成子模块，用于根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；目标数据获取子模块，用于根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
在一个实施例中，各个目标下载任务对应的任务数据是json字符串数据。下载任务生成模块，用于将生成的下载任务的任务数据以json字符串的形式存入异步任务表。
查询信息生成子模块，包括：解析单元，用于解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；查询信息生成单元，用于对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
在一个实施例中，下载任务生成模块，包括：标识确定子模块，用于在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；查询子模块，用于查询异步任务表；下载任务生成子模块，用于当异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务时，生成与下载请求对应的下载任务；提示消息发送子模块，用于当异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务时，向任一客户端返回重复下载提示消息。
在一个实施例中，下载任务异步处理装置还包括：操作请求响应模块，用于响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息；任务状态修改模块，用于根据任务操作信息修改待操作下载任务的任务状态信息。
关于下载任务异步处理装置的具体限定可以参见上文中对于下载任务异步处理方法的限定，在此不再赘述。上述下载任务异步处理装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，如图4所示，提供了一种下载任务异步处理系统，该系统包括下载请求处理模块210、下载任务处理模块220、以及下载数据管理模块230；下载任务处理模块包括一个或多个任务处理节点，任一个任务处理节点配置有定时任务；下载请求处理模块210，用于在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务，将生成的下载任务存入异步任务表；还用于确定异步任务表中的已处理下载任务，将已处理下载任务对应的存储地址发送给已处理下载任务对应的客户端；任一个任务处理节点，用于在本任务处理节点的定时任务触发时，确定与本任务处理节点对应的预设任务处理配置信息，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务，获取各个目标下载任务对应的目标下载数据，将各个目标下载任务对应的目标下载数据存入下载数据管理模块，将各个目标下载任务对应的目标下载数据在下载数据管理模块中的存储地址更新到异步任务表中；下载数据管理模块230，用于存储下载数据；还用于在接收到来自任一客户端的存储地址时，将与存储地址对应的下载数据发送给该任一客户端。
在一个实施例中，预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；任一个任务处理节点在执行从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务的操作时，具体用于：根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务。
在一个实施例中，任一个任务处理节点在执行从多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务的操作时，具体用于：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
在一个实施例中，任一个任务处理节点在执行获取各个目标下载任务对应的目标下载数据的操作时，具体用于：获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据；在一个实施例中，各个目标下载任务对应的任务数据是json字符串数据；下载请求处理模块在执行将生成的下载任务存入异步任务表的操作时，具体用于：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；任一个任务处理节点在执行根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息的操作时，具体用于：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
在一个实施例中，下载请求处理模块在执行在接收到来自任一客户端的下载请求时生成与下载请求对应的下载任务的操作时，具体用于：在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；查询异步任务表；若异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务，生成与下载请求对应的下载任务
若异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务，向任一客户端返回重复下载提示消息。
在一个实施例中，该下载任务异步处理系统还包括下载任务管理模块240。下载任务管理模块240，用于响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息，根据任务操作信息修改待操作下载任务的任务状态信息。
关于下载任务异步处理系统的具体限定可以参见上文中对于下载任务异步处理方法的限定，在此不再赘述。上述下载任务异步处理系统中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，其内部结构图可以如图5所示。该计算机设备包括通过系统总线连接的处理器、存储器、网络接口和数据库。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的数据库用于异步任务表等数据。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种下载任务异步处理方法。
本领域技术人员可以理解，图5中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现以下步骤：
在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务；将生成的下载任务存入异步任务表；在预设定时任务触发时，确定预设任务处理配置信息；从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务；获取各个目标下载任务对应的目标下载数据；将各个目标下载任务对应的目标下载数据存入下载数据管理设备；将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据。
在一个实施例中，预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；处理器执行计算机程序，实现从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务时，还实现以下步骤：根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务。
在一个实施例中，处理器执行计算机程序，实现从多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务时，还实现以下步骤：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
在一个实施例中，处理器执行计算机程序，实现获取各个目标下载任务对应的目标下载数据时，还实现以下步骤：获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
在一个实施例中，各个目标下载任务对应的任务数据是json字符串数据；处理器执行计算机程序，实现将生成的下载任务存入异步任务表时，还实现以下步骤：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；处理器执行计算机程序，实现根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息时，还实现以下步骤：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
在一个实施例中，处理器执行计算机程序，实现在接收到来自任一客户端的下载请求时生成与下载请求对应的下载任务时，还实现以下步骤：在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；查询异步任务表；若异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务，生成与下载请求对应的下载任务；若异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务，向任一客户端返回重复下载提示消息。
在一个实施例中，处理器执行计算机程序时，还实现以下步骤：响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息；根据任务操作信息修改待操作下载任务的任务状态信息。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：在接收到来自任一客户端的下载请求时，生成与下载请求对应的下载任务；将生成的下载任务存入异步任务表；在预设定时任务触发时，确定预设任务处理配置信息；从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务；获取各个目标下载任务对应的目标下载数据；将各个目标下载任务对应的目标下载数据存入下载数据管理设备；将各个目标下载任务对应的目标下载数据在下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从下载数据管理设备下载与各个目标下载任务对应的目标下载数据。
在一个实施例中，预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；计算机程序被处理器执行，从异步任务表中确定与预设任务处理配置信息匹配的多个目标下载任务时，还实现以下步骤：根据任务筛选信息从异步任务表中筛选出多个候选未处理下载任务；从多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务。
在一个实施例中，计算机程序被处理器执行，从多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务时，还实现以下步骤：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将多个候选未处理下载任务进行排序；将多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
在一个实施例中，计算机程序被处理器执行，获取各个目标下载任务对应的目标下载数据时，还实现以下步骤：获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
在一个实施例中，各个目标下载任务对应的任务数据是json字符串数据；计算机程序被处理器执行，将生成的下载任务存入异步任务表时，还实现以下步骤：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；处理器执行计算机程序，根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息时，还实现以下步骤：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
在一个实施例中，计算机程序被处理器执行，在接收到来自任一客户端的下载请求时生成与下载请求对应的下载任务时，还实现以下步骤：在接收到来自任一客户端的下载请求时，确定下载请求对应的下载任务标识和用户标识；查询异步任务表；若异步任务表中不包括与下载任务标识和用户标识对应的未处理下载任务，生成与下载请求对应的下载任务；若异步任务表中包括与下载任务标识和用户标识对应的未处理下载任务，向任一客户端返回重复下载提示消息。
在一个实施例中，计算机程序被处理器执行时，还实现以下步骤：响应来自管理员客户端的任务操作请求，确定任务操作请求对应的待操作下载任务和任务操作信息；根据任务操作信息修改待操作下载任务的任务状态信息。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (10)

  1. 一种下载任务异步处理方法，其特征在于，包括：
    在接收到来自任一客户端的下载请求时，生成与所述下载请求对应的下载任务；将生成的下载任务存入异步任务表；在预设定时任务触发时，确定预设任务处理配置信息；从所述异步任务表中确定与所述预设任务处理配置信息匹配的多个目标下载任务；获取各个目标下载任务对应的目标下载数据；将各个目标下载任务对应的目标下载数据存入下载数据管理设备；
    将各个目标下载任务对应的目标下载数据在所述下载数据管理设备中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从所述下载数据管理设备下载与所述各个目标下载任务对应的目标下载数据。
  2. 如权利要求1所述的下载任务异步处理方法，其特征在于，所述预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；
    所述从所述异步任务表中确定与所述预设任务处理配置信息匹配的多个目标下载任务，包括：根据所述任务筛选信息从所述异步任务表中筛选出多个候选未处理下载任务；从所述多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务；
    优选地，所述从所述多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务，包括：
    确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将所述多个候选未处理下载任务进行排序；将所述多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
  3. 如权利要求1所述的下载任务异步处理方法，其特征在于，所述获取各个目标下载任务对应的目标下载数据，包括：获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据。
    优选地，各个目标下载任务对应的任务数据是json字符串数据；
    所述将生成的下载任务存入异步任务表，包括：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；
    所述根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息，包括：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
  4. 如权利要求1所述的下载任务异步处理方法，其特征在于，
    所述在接收到来自任一客户端的下载请求时生成与所述下载请求对应的下载任务，包括：
    在接收到来自任一客户端的下载请求时，确定所述下载请求对应的下载任务标识和用户标识；查询异步任务表；若所述异步任务表中不包括与所述下载任务标识和用户标识对应的未处理下载任务，生成与所述下载请求对应的下载任务；若所述异步任务表中包括与所述下载任务标识和用户标识对应的未处理下载任务，向所述任一客户端返回重复下载提示消息；
    优选地，所述方法还包括：响应来自管理员客户端的任务操作请求，确定所述任务操作请求对应的待操作下载任务和任务操作信息；根据所述任务操作信息修改所述待操作下载任务的任务状态信息。
  5. 一种下载任务异步处理装置，其特征在于，包括：
    下载任务生成模块，用于在接收到来自任一客户端的下载请求时，生成与所述下载请求对应的下载任务；下载任务存入模块，用于将生成的下载任务存入异步任务表；配置信息确定模块，用于在预设定时任务触发时，确定预设任务处理配置信息；目标任务确定模块，用于从所述异步任务表中确定与所述预设任务处理配置信息匹配的多个目标下载任务；目标数据获取模块，用于获取各个目标下载任务对应的目标下载数据；目标数据存入模块，用于将各个目标下载任务对应的目标下载数据存入下载数据管理模块；地址发送模块，用于将各个目标下载任务对应的目标下载数据在所述下载数据管理模块中的存储地址发送给各个目标下载任务对应的目标客户端，以使各个目标下载任务对应的目标客户端通过其接收到的存储地址从所述下载数据管理模块下载与所述各个目标下载任务对应的目标下载数据。
  6. 一种下载任务异步处理系统，其特征在于，所述系统包括下载请求处理模块、下载任务处理模块、以及下载数据管理模块；所述下载任务处理模块包括一个或多个任务处理节点，任一个所述任务处理节点配置有定时任务；
    所述下载请求处理模块，用于在接收到来自任一客户端的下载请求时，生成与所述下载请求对应的下载任务，将生成的下载任务存入异步任务表；还用于确定所述异步任务表中的已处理下载任务，将所述已处理下载任务对应的存储地址发送给所述已处理下载任务对应的客户端；
    任一个所述任务处理节点，用于在本任务处理节点的定时任务触发时，确定与本任务处理节点对应的预设任务处理配置信息，从所述异步任务表中确定与所述预设任务处理配置信息匹配的多个目标下载任务，获取各个目标下载任务对应的目标下载数据，将各个目标下载任务对应的目标下载数据存入所述下载数据管理模块，将各个目标下载任务对应的目标下载数据在所述下载数据管理模块中的存储地址更新到所述异步任务表中；
    所述下载数据管理模块，用于存储下载数据；还用于在接收到来自任一客户端的存储地址时，将与所述存储地址对应的下载数据发送给所述任一客户端。
  7. 如权利要求6所述的下载任务异步处理系统，其特征在于，所述预设任务处理配置信息包括单次任务处理数量Z和任务筛选信息，Z为大于0的正整数；
    任一个所述任务处理节点在执行从所述异步任务表中确定与所述预设任务处理配置信息匹配的多个目标下载任务的操作时，具体用于：根据所述任务筛选信息从所述异步任务表中筛选出多个候选未处理下载任务；从所述多个候选未处理下载任务中确定Z个候选未处理下载任务作为多个目标下载任务。
    优选地，任一个所述任务处理节点在执行从所述多个候选未处理下载任务中确定N个候选未处理下载任务作为多个目标下载任务的操作时，具体用于：确定各个候选未处理下载任务对应的处理优先级；按照各个候选未处理下载任务对应的处理优先级将所述多个候选未处理下载任务进行排序；将所述多个候选未处理下载任务中，处理优先级最高的前Z个候选未处理下载任务确定为多个目标下载任务。
  8. 如权利要求6所述的下载任务异步处理系统，其特征在于，任一个所述任务处理节点在执行获取各个目标下载任务对应的目标下载数据的操作时，具体用于：
    获取各个目标下载任务对应的任务数据；根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息；根据各个目标下载任务对应的查询信息获取与各个目标下载任务对应的目标下载数据；
    优选地，各个目标下载任务对应的任务数据是json字符串数据；
    所述下载请求处理模块在执行将生成的下载任务存入异步任务表的操作时，具体用于：将生成的下载任务的任务数据以json字符串的形式存入异步任务表；
    任一个所述任务处理节点在执行根据各个目标下载任务对应的任务数据生成各个目标下载任务对应的查询信息的操作时，具体用于：解析各个目标下载任务对应的json字符串数据，得到各个目标下载任务对应的解析结果；对各个目标下载任务对应的解析结果进行拼装处理，得到各个目标下载任务对应的查询信息。
  9. 如权利要求6所述的下载任务异步处理系统，其特征在于，所述下载请求处理模块在执行在接收到来自任一客户端的下载请求时生成与所述下载请求对应的下载任务的操作时，具体用于：在接收到来自任一客户端的下载请求时，确定所述下载请求对应的下载任务标识和用户标识；查询异步任务表；若所述异步任务表中不包括与所述下载任务标识和用户标识对应的未处理下载任务，生成与所述下载请求对应的下载任务；若所述异步任务表中包括与所述下载任务标识和用户标识对应的未处理下载任务，向所述任一客户端返回重复下载提示消息；
    优选地，所述系统还包括下载任务管理模块；所述下载任务管理模块，用于响应来自管理员客户端的任务操作请求，确定所述任务操作请求对应的待操作下载任务和任务操作信息，根据所述任务操作信息修改所述待操作下载任务的任务状态信息。
  10. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至4中任一项所述的方法的步骤。
PCT/CN2021/095194 2020-05-26 2021-05-21 下载任务异步处理方法、装置、系统和存储介质 WO2021238796A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010454003.8A CN111800459A (zh) 2020-05-26 2020-05-26 下载任务异步处理方法、装置、系统和存储介质
CN202010454003.8 2020-05-26

Publications (1)

Publication Number Publication Date
WO2021238796A1 true WO2021238796A1 (zh) 2021-12-02

Family

ID=72806254

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/095194 WO2021238796A1 (zh) 2020-05-26 2021-05-21 下载任务异步处理方法、装置、系统和存储介质

Country Status (2)

Country Link
CN (1) CN111800459A (zh)
WO (1) WO2021238796A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114218156A (zh) * 2021-12-07 2022-03-22 深圳市科脉技术股份有限公司 数据导出方法、装置、计算机设备和存储介质
CN114584622A (zh) * 2022-03-03 2022-06-03 平安普惠企业管理有限公司 批量数据处理方法、装置、计算机设备及存储介质
CN114925135A (zh) * 2022-04-24 2022-08-19 重庆紫光华山智安科技有限公司 数据导出方法、装置、系统、电子设备及存储介质
CN115022303A (zh) * 2022-05-17 2022-09-06 政采云有限公司 一种文件下载方法、装置、设备及介质
CN116561198A (zh) * 2023-07-12 2023-08-08 成方金融科技有限公司 一种供应链金融的统计数据下载方法和装置
CN116614551A (zh) * 2023-07-17 2023-08-18 北京海科融通支付服务有限公司 基于模板的多任务异步下载方法、系统、设备及介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111800459A (zh) * 2020-05-26 2020-10-20 苏宁云计算有限公司 下载任务异步处理方法、装置、系统和存储介质
CN112256417B (zh) * 2020-10-23 2024-03-29 上海豹云网络信息服务有限公司 一种数据请求处理方法、装置及计算机可读存储介质
CN112433832A (zh) * 2020-11-17 2021-03-02 北京沃东天骏信息技术有限公司 一种任务触发方法、装置、设备及计算机可读存储介质
CN113032040B (zh) * 2021-03-15 2023-07-28 北京百度网讯科技有限公司 用于处理任务的方法、装置、设备、介质和产品
CN114756573B (zh) * 2022-06-16 2022-10-04 恒生电子股份有限公司 数据处理方法、装置及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080189350A1 (en) * 2007-02-06 2008-08-07 Vasa Prakash Devices, systems, and methods for providing data
CN103533002A (zh) * 2012-07-05 2014-01-22 阿里巴巴集团控股有限公司 一种数据处理方法和系统
CN110555068A (zh) * 2018-05-14 2019-12-10 北京京东尚科信息技术有限公司 数据导出方法和装置
CN111104448A (zh) * 2019-12-17 2020-05-05 深圳前海环融联易信息科技服务有限公司 大数据量Excel文件导出方法、装置、计算机设备及存储介质
CN111800459A (zh) * 2020-05-26 2020-10-20 苏宁云计算有限公司 下载任务异步处理方法、装置、系统和存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103207785B (zh) * 2013-04-23 2016-08-31 北京奇虎科技有限公司 数据下载请求的处理方法、装置及系统
CN104881307B (zh) * 2015-05-15 2018-09-04 广东小天才科技有限公司 下载实现方法和装置
US9948704B2 (en) * 2016-04-07 2018-04-17 International Business Machines Corporation Determining a best fit coordinator node in a database as a service infrastructure
CN109462650A (zh) * 2018-11-14 2019-03-12 深圳市小牛普惠投资管理有限公司 数据文件下载方法、装置、计算机设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080189350A1 (en) * 2007-02-06 2008-08-07 Vasa Prakash Devices, systems, and methods for providing data
CN103533002A (zh) * 2012-07-05 2014-01-22 阿里巴巴集团控股有限公司 一种数据处理方法和系统
CN110555068A (zh) * 2018-05-14 2019-12-10 北京京东尚科信息技术有限公司 数据导出方法和装置
CN111104448A (zh) * 2019-12-17 2020-05-05 深圳前海环融联易信息科技服务有限公司 大数据量Excel文件导出方法、装置、计算机设备及存储介质
CN111800459A (zh) * 2020-05-26 2020-10-20 苏宁云计算有限公司 下载任务异步处理方法、装置、系统和存储介质

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114218156A (zh) * 2021-12-07 2022-03-22 深圳市科脉技术股份有限公司 数据导出方法、装置、计算机设备和存储介质
CN114584622A (zh) * 2022-03-03 2022-06-03 平安普惠企业管理有限公司 批量数据处理方法、装置、计算机设备及存储介质
CN114925135A (zh) * 2022-04-24 2022-08-19 重庆紫光华山智安科技有限公司 数据导出方法、装置、系统、电子设备及存储介质
CN115022303A (zh) * 2022-05-17 2022-09-06 政采云有限公司 一种文件下载方法、装置、设备及介质
CN116561198A (zh) * 2023-07-12 2023-08-08 成方金融科技有限公司 一种供应链金融的统计数据下载方法和装置
CN116614551A (zh) * 2023-07-17 2023-08-18 北京海科融通支付服务有限公司 基于模板的多任务异步下载方法、系统、设备及介质
CN116614551B (zh) * 2023-07-17 2023-09-22 北京海科融通支付服务有限公司 基于模板的多任务异步下载方法、系统、设备及介质

Also Published As

Publication number Publication date
CN111800459A (zh) 2020-10-20

Similar Documents

Publication Publication Date Title
WO2021238796A1 (zh) 下载任务异步处理方法、装置、系统和存储介质
KR102541295B1 (ko) 온-디맨드 네트워크 코드 실행 시스템에서의 운영 체제 커스터마이제이션
US11422853B2 (en) Dynamic tree determination for data processing
US10225145B2 (en) Method and device for updating client
US10114682B2 (en) Method and system for operating a data center by reducing an amount of data to be processed
WO2020253079A1 (zh) 基于Jmeter的分布式性能测试方法、装置、设备及存储介质
US20170286143A1 (en) Processing pre-existing data sets at an on demand code execution environment
US20170286156A1 (en) Generating data streams from pre-existing data sets
US8024744B2 (en) Method and system for off-loading user queries to a task manager
CN111930770A (zh) 数据查询方法、装置及电子设备
CN112463290A (zh) 动态调整计算容器的数量的方法、系统、装置和存储介质
US9342369B2 (en) System and method for increasing throughput of a PaaS system
WO2021022714A1 (zh) 跨区块链节点的消息处理方法及装置、设备、介质
CN113656157A (zh) 一种分布式任务调度方法、装置、存储介质及电子设备
CN112243032A (zh) 基于访问网关的服务调用方法及其系统
WO2022257247A1 (zh) 数据处理方法、装置及计算机可读存储介质
US11816119B2 (en) System and methods for querying and updating databases
CN112579319A (zh) 一种基于LRU Cache优化的服务调用方法及装置
US11144359B1 (en) Managing sandbox reuse in an on-demand code execution system
CN110955460B (zh) 一种服务进程启动方法、装置、电子设备和存储介质
CN110365809B (zh) 分布式服务器地址配置系统及方法
CN111008146A (zh) 对云主机安全进行测试的方法及系统
US20210377718A1 (en) Pattern affinity for discovery
CN110489694A (zh) 用于处理高并发请求的方法和系统
CN113055498B (zh) 数据源访问方法、装置、存储介质及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21813630

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21813630

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21813630

Country of ref document: EP

Kind code of ref document: A1