CN117149588A

CN117149588A - Task processing method and related equipment

Info

Publication number: CN117149588A
Application number: CN202311122591.5A
Authority: CN
Inventors: 余俊涛
Original assignee: CMB Yunchuang Information Technology Co Ltd
Current assignee: CMB Yunchuang Information Technology Co Ltd
Priority date: 2023-09-01
Filing date: 2023-09-01
Publication date: 2023-12-01

Abstract

The embodiment of the application provides a task processing method and related equipment, which are used for processing tasks in batches so as to improve task processing efficiency as much as possible. The method of the embodiment of the application comprises the following steps: receiving a task processing request initiated by a user at a client, and generating a task identification code corresponding to the task processing request; the task processing request is a processing request generated by a user based on the selected target task item at the client side; establishing communication connection between the client and the server, and sending a task identification code, a target task item and a task processing request to the server so that the server records the current processing progress of the target task item according to the task identification code; the current processing progress is the processing progress obtained after the server side processes the target task item according to the task processing request; and receiving the current processing progress corresponding to the task identification code, and displaying the current processing progress on the client.

Description

Task processing method and related equipment

Technical Field

The embodiment of the application relates to the technical field of data processing, in particular to a task processing method and related equipment.

Background

The common business system has the functions of task query and task processing, the task list query and task batch processing requirements are correspondingly generated, some simple batch task processing may be small in quantity, and business logic is simple, so that the requirements can be met by using a common batch processing scheme. However, as the traffic volume increases dramatically, the complexity and relevance of the traffic processing increases. When a user is faced with a large-batch task scheduling process, if a conventional batch processing scheme is adopted, problems may be faced, such as overtime caused by too long response time of a batch request server, overload and blocking, breakdown and the like of the server caused by too large batch task amount.

Therefore, there is a need for a method that can reasonably utilize the resources of the server cluster to process efficient batch tasks, so as to ensure the normal processing of task processing as much as possible.

Disclosure of Invention

The embodiment of the application provides a task processing method and related equipment, which are used for processing tasks in batches so as to improve task processing efficiency as much as possible.

The first aspect of the embodiment of the application provides a task processing method, which is applied to a client and comprises the following steps:

receiving a task processing request initiated by a user at the client, and generating a task identification code corresponding to the task processing request; the task processing request is a processing request generated by the user on the basis of the selected target task item by the client;

establishing communication connection between the client and the server, and sending the task identification code, the target task item and the task processing request to the server so that the server records the current processing progress of the target task item according to the task identification code; the current processing progress is a processing progress obtained after the server side processes the target task item according to the task processing request;

And receiving the current processing progress corresponding to the task identification code, and displaying the current processing progress on the client.

Optionally, after receiving the current processing progress corresponding to the task identification code, the method further includes:

judging whether the value of the current processing progress is larger than the value of the processing progress displayed by the current page of the client;

if yes, updating the processing progress displayed by the current page to the current processing progress so as to display the current processing progress.

Optionally, the method further comprises:

if the current processing progress meets the target processing progress, closing the target processing progress displayed on the current page of the client;

inquiring a task processing result of the target task item at the server according to the task identification code, and displaying the task processing result at the current page of the client.

A second aspect of the embodiment of the present application provides a task processing method, applied to a server, including:

receiving a task identification code, a target task item and a task processing request which are transmitted by a client through communication connection; the task processing request is a processing request initiated by a user at the client and aimed at the target task item, and the task identification code is an identification code corresponding to the task processing request;

Executing the target task item according to the task processing request, and acquiring the current processing progress corresponding to the target task item;

and sending all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item.

Optionally, the server includes a buffer, and after receiving the task identifier, the target task item, and the task processing request transmitted by the client through the communication connection, the method further includes:

and storing the target task item into the buffer based on the task identification code, and recording the processing progress of the target task item as an initial processing progress.

Optionally, the server includes a repeater, an executor and a buffer, where the repeater is used for data transmission, and the executor is used for executing a task item, and the executing the target task item according to the task processing request includes:

dividing the task processing request into a plurality of subtask processing requests; any subtask processing request corresponds to any target task item;

distributing all the subtask processing requests to the executor through the forwarder so that the executor executes any one of the target task items according to the subtask processing requests;

When the execution of the target task item is completed, recording the task processing result of the target task item as completed, and storing the task processing result, the task identification code and the subtask processing request into the buffer; and the task processing result, the task identification code and the subtask processing request have an association relation.

Optionally, the acquiring the current processing progress corresponding to the target task item includes:

acquiring the number of processing tasks corresponding to the task identification code; wherein the number of processing tasks is the number of completed task processing results corresponding to the subtask processing request;

and calculating the current processing progress according to the number of the processing tasks and the number of the target tasks corresponding to the target task items.

Optionally, after the sending all the current processing progress corresponding to the task identification code to the client, the method further includes:

receiving a result query request sent by the client, and acquiring all task processing results of the target task item corresponding to the task identification code according to the result query request;

And sending all the task processing results to the client.

Optionally, the server includes a server, and before the receiving the task identification code, the target task item and the task processing request transmitted by the client through the communication connection, the method further includes:

establishing communication connection between the client and the server;

the receiving the task identification code, the target task item and the task processing request transmitted by the client through the communication connection comprises the following steps:

storing the task identification code to any server, and determining that any server is a target server;

acquiring the current processing progress corresponding to the target task item, including:

the target server obtains the current processing progress of the target task item;

judging whether the identification code of the current processing progress corresponds to the task identification code or not;

if yes, executing the step of sending all the current processing progress corresponding to the task identification code to the client through the communication connection;

if not, determining that the target server does not exist in the server, and stopping executing the step of sending all the current processing progress corresponding to the task identification code to the client.

when the current processing progress meets the target processing progress, disconnecting the communication connection between the client and the server;

and when the communication connection is disconnected, clearing the task identification code cached by the server and the current processing progress corresponding to the task identification code.

A third aspect of an embodiment of the present application provides a task processing system, applied to a client, including:

the receiving unit is used for receiving a task processing request initiated by a user at the client and generating a task identification code corresponding to the task processing request; the task processing request is a processing request generated by the user on the basis of the selected target task item by the client;

the establishing unit is used for establishing communication connection between the client and the server and sending the task identification code, the target task item and the task processing request to the server so that the server records the current processing progress of the target task item according to the task identification code; the current processing progress is a processing progress obtained after the server side processes the target task item according to the task processing request;

The receiving unit is further configured to receive the current processing progress corresponding to the task identifier, and display the current processing progress on the client.

A third aspect of the embodiment of the present application provides a method for executing the task processing described in the first aspect.

A fourth aspect of the present application provides a task processing system, applied to a client, including:

the receiving unit is used for receiving the task identification code, the target task item and the task processing request which are transmitted by the client through communication connection; the task processing request is a processing request initiated by a user at the client and aimed at the target task item, and the task identification code is an identification code corresponding to the task processing request;

the execution unit is used for executing the target task item according to the task processing request and acquiring the current processing progress corresponding to the target task item;

and the sending unit is used for sending all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item.

The fourth aspect of the embodiment of the present application provides a task processing method for executing the first aspect.

A fifth aspect of an embodiment of the present application provides a task processing device, including:

the device comprises a central processing unit, a memory, an input/output interface, a wired or wireless network interface and a power supply;

the memory is a short-term memory or a persistent memory;

the central processor is configured to communicate with the memory and to execute instruction operations in the memory to perform the task processing method of the first or second aspect.

A sixth aspect of an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the task processing method according to the first aspect or the second aspect.

From the above technical solutions, the embodiment of the present application has the following advantages: according to the task processing method disclosed by the embodiment of the application, a task identification code, a target task item and a task processing request which are transmitted by a client through communication connection are received; the task processing request is a processing request which is initiated by a user at a client and aims at a target task item, and the task identification code is an identification code corresponding to the task processing request; executing the target task item according to the task processing request, and acquiring the current processing progress corresponding to the target task item; and finally, transmitting all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item. Therefore, the target task items can be processed in batches according to the task processing request of the user, and meanwhile, the target task items are all under the same task identification code, so that the server side can reasonably utilize the current server resources to process the target task items in parallel, and the task processing efficiency is improved as much as possible.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments described in the present application, and other drawings may be obtained according to these drawings for those of ordinary skill in the art.

FIG. 1 is a schematic diagram of a task processing system according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a task processing method applied to a client according to an embodiment of the present application;

FIG. 3 is a flowchart of another task processing method applied to a client according to an embodiment of the present application;

fig. 4 is a schematic flow chart of a task processing method applied to a server according to an embodiment of the present application;

FIG. 5 is a flowchart of another task processing method applied to a server according to an embodiment of the present application;

FIG. 6 is a flow chart of a task processing method according to an embodiment of the present application;

FIG. 7 is an interface diagram of a batch task selection according to an embodiment of the present application;

FIG. 8 is an interface diagram of task progress transmission according to an embodiment of the present application;

FIG. 9 is an interface diagram of a task progress bar according to an embodiment of the present application;

FIG. 10 is a diagram of a feedback interface for task results according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a task processing system applied to a client according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a task processing system applied to a server according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a task processing device according to an embodiment of the present application.

Detailed Description

The common business system has some task inquiring and processing functions, and correspondingly generates the demands of inquiring and batch choosing business task lists for batch processing, and some simple batch task processing can be less in quantity, and business logic is simple, so that the demands can be met by using a general batch processing scheme. However, as the traffic volume increases, the complexity and relevance of the traffic processing increases. When a user is faced with a large-batch task scheduling process, if a general batch processing scheme is adopted again, problems may be faced, such as overtime caused by too long response time of a batch request server, overload and blocking, breakdown and the like of the server caused by too large batch task quantity.

Typical batch processing schemes are as follows:

and the first server directly circularly processes after receiving the batch task request. The server directly processes tasks needing batch processing circularly in the request main thread, and processes the second task after the first task is processed, and so on. The method can simulate the multiple single processing task summarization of the user, but only uses the single node resource of the system, and only uses single thread to process the task, so that the performance is poor. If the transaction of the database is at the unified entrance, the situations that the batch task processing transactions are mutually influenced, the batch task fails, all the batch tasks fail, and the like exist. If the average time for processing one task is 0.1 seconds, 1000 tasks are 100 seconds, which easily results in a request timeout.

And secondly, after receiving the batch task request, the server utilizes multithreading to process. The server gathers tasks to be processed in batches, starts sub-threads through a multithreading technology, processes one task by each sub-thread, collects task processing results, and then synchronously waits for all task processing results to be returned to a user after all task processing is completed by the main thread. The method not only can simulate a user to process tasks for a plurality of times by one stroke, but also can process tasks in parallel by utilizing the CPU thread number of the single server node. The multithreading process is relatively simple and cyclic processing tasks, the resource parallel processing tasks of the current server node are reasonably utilized, but only the resource of one node is utilized for processing tasks, when the number of batch tasks is too large and exceeds the maximum number of threads of a thread pool, the tasks are difficult to expand, the risk of overtime of interfaces exists, the memory and the CPU of the current server node are caused to surge, and the server is unstable and even down.

The two common batch processing schemes cannot reasonably utilize the resources of the system cluster when processing a large amount of data simultaneously, and the user experience is deficient. Therefore, a scheme is needed that resources of a server cluster can be reasonably utilized to process high-efficiency batch tasks, service availability can be guaranteed when peak clipping and valley filling are performed on a large number of tasks, a user can see task processing progress in real time, and finally, after all tasks are processed, the user can see processing results of each task in the first time.

The following description of the embodiments of the present application will be made with reference to the accompanying drawings, in which it is evident that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

Referring to fig. 1, fig. 1 is a schematic diagram of a task processing system according to an embodiment of the present application. Specifically, the task processing system includes a client 101, a gateway 102, and a server 103. The client 101 is communicatively connected to the server 103 through the gateway 102. In one specific scheme, the gateway 102 is used for transferring information or data of the client 101 and the server 103, so in the scheme of the present application, the gateway 102 may not be provided, and the corresponding function may be implemented in the server 103 or the client 101, which is not limited herein.

As shown in fig. 1, in particular, the system architecture includes a client 101, a server 103, and a gateway 102, where the client 101 may include various terminal devices. The server 103 may be an independent physical server, a server cluster 1032 or a distributed system formed by a plurality of physical servers, or a cloud server that provides a cloud database, a cloud service, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a CDN, and basic cloud computing services such as a big data and an artificial intelligent platform. In the present embodiment, the server 103 is mainly described in detail as the server cluster 1032. In a specific embodiment, the server 103 may further include a gateway 102, a buffer, a repeater, an executor, and the like. Specifically, the executor is a server cluster 1032, which is simply a cluster, and is a server system, and is connected by a set of loosely integrated server software and/or hardware to perform computing work in a highly tight manner. In a sense they can be seen as a server. The individual servers in the cluster 1032 system are commonly referred to as node pods (pod 1-pod4 in fig. 1, the corresponding mullions can be understood as node threads or consuming threads), typically connected by a local area network, but there are other possible ways of connecting. The cluster 1032 servers are typically used to improve the computation speed and/or reliability of individual servers. Typically, the servers of cluster 1032 are more cost-effective than a single server, such as a workstation or a super server. Cluster 1032 is a group of independent servers that are combined into a single group through a network connection to accomplish a task together. In other words, clusters 1032 are a group of independent servers that form a server system through a high-speed network, and each node 1032 is an independent server running its own process. For network users, the back end of the website is a single system, and the system resources and the system services are provided for the users in a cooperative way. And the transponders therein may be understood as gateways 102 or rabitmq 1031. Meanwhile, the RabbitMQ1031 is a message queue middleware, has an efficient and reliable message asynchronous transfer mechanism, is mainly used for data communication among different systems and a transfer buffer is Redis, wherein Redis is a key-value type cache service, and all data are stored in a memory.

The client 101 may be a smart terminal such as a smart phone, tablet, notebook, desktop, palm, mobile internet device (MID, mobile internet device), wearable device (e.g., smart watch, smart bracelet, etc.), smart computer, smart car, etc. Wherein the server 103 may establish a communication connection with a terminal device in the client 101. The communication connection is not limited to a connection manner, and may be directly or indirectly connected through a wired communication manner, or may be directly or indirectly connected through a wireless communication manner, and the like, and may be specifically determined according to an actual application scenario, and the present application is not limited herein. Meanwhile, for convenience of description, the description of the client 101 or the server 103 and the like will not be repeated.

For convenience in describing the interaction of the system architecture in fig. 1 in detail, please refer to fig. 2, fig. 2 is a flow chart of a task processing method applied to a client disclosed in an embodiment of the present application. Including steps 201-203.

201. And receiving a task processing request initiated by a user at the client, and generating a task identification code corresponding to the task processing request.

The client can receive a task processing request initiated by a user at the front end and generate a task identification code corresponding to the task processing request. The task processing request is a processing request generated by a target task item selected by a user at a client.

In one specific embodiment, the user may batch pick up tasks that need to be processed in batches at the front-end interface of the client. Thus, the client may generate relevant task processing requests based on the user-selected tasks. Specifically, referring to fig. 7, fig. 7 is an interface diagram of batch task selection according to an embodiment of the present application. It will be appreciated that the user selects a task flow with different ID identifications through the task selection interface of the client, thereby generating a relevant task processing request. The client may then generate a unique code as the batch number for the entire batch job processing request. The batch number is the task identification code described in the above. For ease of understanding, the following is illustrative. Specifically, if the user selects 200 tasks, the lot numbers of the 200 tasks are all a.

202. And establishing communication connection between the client and the server, and sending the task identification code, the target task item and the task processing request to the server so that the server records the current processing progress of the target task item according to the task identification code.

Based on the task processing request of the user, the client can establish communication connection with the server according to the task identification code. And the task identification code, the target task item and the task processing request are sent to the server through the communication connection. Thus, the server can record the current processing progress of the target task item according to the task identification code. For convenience of description, detailed description will be made later in fig. 4 and 5.

In one specific embodiment, the client may establish a WebSocket connection with the server according to the lot number, that is, the task identifier, and the WebSocket connection may be further used to monitor and receive, in real time, the task processing progress of the lot number transmitted by the server. And then, the client formally initiates a large-batch task processing request by taking the large-batch task and the batch number as request parameters. And sending the related task identification code, the target task item and the task processing request to the server. It should be noted that the WebSocket connection is the new and persistent protocol next to HTML5 (WebSocket protocol is essentially a tcp-based protocol). The method can realize full duplex communication between the client and the server, can better save server resources and bandwidth and achieves the purpose of real-time communication.

Based on the foregoing embodiment, in another specific embodiment, the client may specifically establish WebSocket connection with a specific server in the server cluster. Therefore, the client side only needs to be in communication connection with a specific server, and bandwidth occupation is reduced.

203. And receiving the current processing progress corresponding to the task identification code, and displaying the current processing progress on the client.

After the client establishes WebSocket connection, the waiting server transmits the current processing progress under the task identification code to the client through the WebSocket connection. Thus, the current processing progress is displayed on the client. It is to be understood that the current processing progress is a processing progress obtained after the server side processes the target task item according to the related task processing request.

In one embodiment, referring to fig. 9, fig. 9 is an interface diagram of a task progress bar according to an embodiment of the present application. After receiving the current processing progress of the target task item under the task identification code, the client can render the progress of the progress bar in real time, so that the progress bar is displayed in a front-end interface of the client.

According to the task processing method disclosed by the embodiment, the client side sends the selected target task item to the server side according to the task processing request of the user, and meanwhile, the client side can display the processing progress of the target task item in real time, so that the user experience is improved.

For convenience in describing the task processing method applied to the client in detail in fig. 2, please refer to fig. 3, and fig. 3 is a flow chart of another task processing method applied to the client according to an embodiment of the present application. Including steps 301-306.

301. And receiving a task processing request initiated by a user at the client, and generating a task identification code corresponding to the task processing request.

302. And establishing communication connection between the client and the server, and sending the task identification code, the target task item and the task processing request to the server so that the server records the current processing progress of the target task item according to the task identification code.

303. And receiving the current processing progress corresponding to the task identification code, and displaying the current processing progress on the client.

In this embodiment, steps 301 to 303 are similar to steps 201 to 203 in fig. 2, and detailed descriptions thereof are omitted herein. However, it should be noted that, in one specific embodiment, the client receives the progress of the task processing corresponding to the current task identifier.

304. And judging whether the value of the current processing progress is larger than the value of the processing progress displayed on the current page of the client. If the value of the current processing progress is greater than the value of the processing progress displayed on the current page of the client, step 305 is executed.

And after the client receives the current processing progress, judging the value of the current processing progress and the value of the processing progress displayed on the current page. If the current progress value is greater than the current page display progress value, step 305 is executed. If the number is smaller than the preset number, no processing is performed.

In one specific embodiment, the client may receive the processing progress of the target task item forwarded by the server in real time, and display the current processing progress of the target task item on the front page, so that if the client continues to receive the processing progress in the subsequent step, the client performs the above-mentioned judgment action.

Based on the above embodiment, in another specific embodiment, since the progress bar is a growing process, the backend server performs asynchronous processing in order to pursue performance. A scenario may occur where the client first receives 60% of the progress, renders 60% of the progress bar at this time, and then suddenly receives 50% of the progress of the process, and if the progress bar is rendered, the progress bar is retracted, so no process is performed.

305. And updating the processing progress displayed by the current page into the current processing progress so as to display the current processing progress.

When the processing progress of the current page display is larger than the current processing progress, updating the processing of the current page display to the current processing progress, and displaying the current processing progress.

306. If the current processing progress meets the target processing progress, closing the target processing progress displayed on the current page of the client, inquiring the task processing result of the target task item at the server according to the task identification code, and displaying the task processing result on the current page of the client.

And finally, when the current processing progress meets the target processing progress, the client can close the target processing progress displayed on the current page, inquire the task processing result of the target task item at the server according to the task identification code, and display the task processing result on the current page of the client.

In one particular embodiment, the target process schedule may be 100% process schedule. Specifically, when the progress of task processing corresponding to the target identification code (current batch number) of the client page reaches 100%, the WebSocket connection established with the server is closed, and the progress bar is displayed and closed. After the progress bar of the client is displayed and closed, the client requests the server to inquire the task processing result of the batch task (namely the target task item) of the corresponding batch by taking the batch number as a parameter, the server inquires the database by taking the batch number, returns the processing result of the task of the current batch, and the client renders the processing result returned by the server in the page. Specifically, referring to fig. 10, fig. 10 is a feedback interface diagram of a task result according to an embodiment of the present application. When the progress of the client progress bar reaches 100%, the overall result feedback of batch tasks can be inquired.

According to the task processing method disclosed by the embodiment, the current processing progress and the task processing result of the target task item can be perceived in real time, and meanwhile, when the current processing progress meets the target processing progress, the communication connection with the WebSocket of the server side can be closed, so that the stability of the server cluster is maintained, the user experience is improved, and the feasibility of the scheme is improved.

Referring to fig. 4, fig. 4 is a flow chart of a task processing method applied to a server according to an embodiment of the present application. Including steps 401-403.

401. And receiving the task identification code, the target task item and the task processing request transmitted by the client through the communication connection.

After the communication connection is established between the server and the client, the task identification code, the target task item and the task processing request transmitted by the client through the communication connection can be received. The task identifier, the target task item, and the task processing request may be described above with reference to step 201 in fig. 2, which is not repeated here.

402. And executing the target task item according to the task processing request, and acquiring the current processing progress corresponding to the target task item.

Based on the above step 401, the server may execute the corresponding target task item according to the task processing request, thereby obtaining the current processing progress corresponding to the target task item.

In one specific embodiment, the server may subdivide all task processing requests to be processed, so as to distribute each subtask processing request to each thread in the server cluster to process a corresponding target task item, and it is not easy to understand that in this embodiment, a single thread may process any target task item. Thus, after the thread starts to execute a specific single target task item, the relevant processing progress can be obtained. Finally, the current processing progress of the target task item is obtained.

403. And sending all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item.

After all the current processing progress corresponding to the task identification code is obtained, the current processing progress can be sent to the client, so that the client displays the current processing progress of the target task item.

In one specific embodiment, the current processing progress may be understood as a ratio of the number of processed tasks to the total number of tasks of the target task item, so as to calculate the progress of task processing under the task identification code according to the ratio.

Based on the above embodiment, the server may send the processing progress of the target task item to the client in real time through WebSocket connection. The client can render the current processing progress and display the relevant processing progress to the user.

According to the task processing method disclosed by the embodiment, a task identification code, a target task item and a task processing request which are transmitted by a client through communication connection are received; the task processing request is a processing request which is initiated by a user at a client and aims at a target task item, and the task identification code is an identification code corresponding to the task processing request; executing the target task item according to the task processing request, and acquiring the current processing progress corresponding to the target task item; and finally, transmitting all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item. Therefore, the target task items can be processed in batches according to the task processing request of the user, and meanwhile, the target task items are all under the same task identification code, so that the server side can reasonably utilize the current server resources to process the target task items in parallel, and the task processing efficiency is improved as much as possible.

For convenience in describing the task processing method described in fig. 4 in detail, please refer to fig. 5, fig. 5 is a flow chart of another task processing method applied to a server disclosed in an embodiment of the present application. Including steps 501-509.

501. And establishing communication connection between the client and the server.

When the client and the server perform information interaction, webSocket connection between the client and the server needs to be established.

In one specific embodiment, the client may establish a WebSocket connection with a specific server in the server. In other words, the front end of the client may establish a WebSocket connection with any server node pod. Instead of all server nodes pod in the server cluster establishing connections, the specific one is not limited here. It should be noted in advance that the server in this embodiment includes a server cluster, that is, the number of servers is not limited in this embodiment.

502. And receiving the task identification code, the target task item and the task processing request transmitted by the client through the communication connection.

Step 502 of this embodiment is similar to step 401 in fig. 4, and detailed description thereof is omitted herein.

503. And storing the target task item into a buffer based on the task identification code, and recording the processing progress of the target task item as the initial processing progress.

When the WebSocket connection between the server and the client is successfully established, the task identification code and the related target task item can be stored in the buffer, and the processing progress of the target task item is recorded as the initial processing progress.

In one specific embodiment, the server may store the lot number information (i.e. the task identification prayer) in a local cache (dis) of the current server pod, and after receiving the related target task item, may record the current processing progress as 0 according to the lot number, and also store the current processing progress into the dis. It is to be understood that, in the present embodiment, the initial processing progress is 0%.

Based on the above embodiment, in another specific embodiment, the server may further store the task identifier code in the above to any server in the server cluster, so that the server may be determined as the target server.

504. Dividing the task processing request into a plurality of subtask processing requests, and distributing all the subtask processing requests to the executor through the forwarder so that the executor executes any target task item according to the subtask processing requests.

The server divides the task processing request into a plurality of subtask requests, and distributes all the subtask processing requests to the executor through the relevant forwarder, so that the executor executes any target task item according to the subtask processing requests. It will be appreciated that subtask processing requests are associated with any target task item.

In one specific embodiment, the server subdivides all the large-batch task processing requests to be processed into sub-task requests, packages the sub-task requests into messages, and sends the messages to the RabbitMQ Broker by designating Direct Exchange and Binding keys. Then, the RabbitMQ Broker finds a consumer Queue through Direct Exchange and Binding Key, sends a specific subtask message to the Queue, and one of the consumer threads of the Queue performs a message consumption processing task. Among them, it is understood that Direct exchange is a switch name of RabbitMQ, which can support multiple servers binding to this switch through one and the same queue. If a message is sent through this switch, only one of the queue's consumers can receive the message. For example, there are 100 batch tasks, and the 1 st task is passed to the switch, which may be assigned to the server 2 for processing, and the 2 nd task is passed to the switch, which may be assigned to the server 10 for processing.

505. When the execution of the target task item is completed, recording the task processing result of the target task item as completed, and storing the task processing result, the task identification code and the subtask processing request into a buffer.

After the server receives the relevant target task item, execution of the target task item is started. After the execution is completed, the task processing result of the target task item is recorded as completed, and the task processing result, the task identification prayer and the subtask processing request are stored in the Redis.

In one particular embodiment, based on step 504 described above, the consumer thread begins executing particular single task processing logic upon receipt of the message. It will be appreciated that one thread may only execute one target task item at a time, and different threads may execute different target task items in parallel. After the task processing is completed, recording the processing result of the task, and writing the information such as the task processing result, the task identification code (batch number), the subtask processing request and the like into a database (Redis).

Based on the above embodiment, when recording the processing result, recording can be performed by mysql statement.

506. And acquiring the number of processing tasks corresponding to the task identification codes, and calculating the current processing progress according to the number of processing tasks and the number of target tasks corresponding to the target task items.

The server side can acquire the number of processing tasks corresponding to the task identification codes in real time, and then calculates the current processing progress according to the number of processing tasks and the number of target tasks corresponding to the target task items. It is to be understood that the number of processing tasks is the number of completed task processing results corresponding to the subtask processing request.

In one specific embodiment, the server may use the task identifier as a key, increase the number of completed tasks under the task identifier by one through Redis, and synchronously acquire the number of target task items that are currently processed and completed. And thus the progress of task processing under the current task identification code is calculated by the total number of the processed tasks/the total number of the tasks. In other words, the ratio of the number of processed completed tasks to the total number of target task items is taken as the current processing progress. Then, the server sends the current processing progress to the RabbitMQ. Specifically, the server packages the task processing progress corresponding to the current batch number into a message, and designates the Fanout Exchange to send to the RabbitMQ Broker. The RabbitMQ Broker finds all Queue queues binding the exchanger through the Fanout Exchange, and the queues send progress messages of task processing corresponding to the current batch number to the consumer node pod.

Based on the above embodiment, in another specific embodiment, the server side further determines whether the identifier of the current processing progress corresponds to the task identifier. If yes, go to step 507; if not, the execution of the subsequent steps is aborted.

Specifically, after the consumer node receives the current processing progress corresponding to the current batch number (task identifier code), it is determined by the batch number whether there is a corresponding batch number in the local cache of the current consumer node pod, if so, the current consumer node pod is the server node pod that establishes a WebSocket connection with the front end of the client, that is, the node of the target server may execute step 507. If not, the method is directly finished.

507. And sending all the current processing progress corresponding to the task identification code to the client so that the client displays the current processing progress of the target task item.

The server may send the current processing progress corresponding to the task identification code to the client, thereby causing the client to display the current processing progress of the target task item.

In one specific embodiment, the target server in the server side sends the current processing progress corresponding to the task identification code to the client side through WebSocket connection. Thus, the client can display the current processing progress of the target task item in real time. For ease of understanding, please refer to fig. 8, fig. 8 is an interface diagram of task progress transmission according to an embodiment of the present application. The server processes the task and can transmit the task progress to the client through the WebSocket in real time.

508. And receiving a result query request sent by the client, and acquiring all task processing results of the target task item corresponding to the task identification code according to the result query request so as to send all task processing results to the client.

When the progress bar of the client is displayed and closed, the client can send a result query request to the server, so that the server can acquire the task processing result of the target task item corresponding to the task identification code according to the result query request, and then the task processing result is returned to the client.

In one specific embodiment, the client may request the server to query a large batch of task processing results of the corresponding batch by using the batch number (task identification code) as a parameter, the server returns the processing results of the task of the current batch by using the batch number query database (Redis), and the client renders the processing results returned by the server in the page.

509. When the current processing progress meets the target processing progress, the communication connection between the client and the server is disconnected, and the task identification code cached by the server and the current processing progress corresponding to the task identification code are cleared.

When the current processing progress of the client meets the target processing progress, the client closes the WebSocket connection with the server and clears the task identification code cached by the server and the current processing progress corresponding to the task identification code.

In one specific embodiment, when the progress of task processing corresponding to the current batch number of the progress bar of the client page reaches 100%, the WebSocket connection established with the server is closed, and the progress bar is displayed for closing. Therefore, after the server receives the WebSocket closing signal, the information of the batch number in the local cache is cleared, and the progress information of the batch number in the Redis is cleared.

It should be understood that the specific execution sequence of steps 508 and 509 is not limited in this embodiment.

By the task processing method in the embodiment, rabbitMQ serves as a task distributor to play a role in peak clipping and valley filling, so that an unlimited task is prevented from being requested to a server at the same time, and the stability of a server cluster is maintained. Meanwhile, the scheme fully utilizes the CPU computing power of each node of the server cluster, greatly improves the efficiency of batch task processing, and simultaneously can cope with the scene of larger batch tasks through the expansion of the server cluster because the tasks are scattered to each server node in the cluster for processing. Furthermore, the Websocket is used as a bridge between the client and the server, so that the perception of the batch task processing progress of the user can be well ensured, and the problem that a large number of tasks directly request the server to wait for response overtime can be avoided.

Based on the task processing method described in fig. 2 to fig. 4, the following describes the interaction flow of the client and the server in detail, please refer to fig. 6, and fig. 6 is a flow interaction diagram of a task processing method according to an embodiment of the present application. Including steps 601-612.

601. The client receives a task processing request initiated by a user at the client and generates a task identification code corresponding to the task processing request.

602. And the client sends the task identification code, the target task item and the task processing request to the server.

Steps 601 to 602 in this embodiment are similar to steps 301 to 302 in fig. 3, and detailed descriptions thereof are omitted herein.

603. The service end stores the target task item into the buffer based on the task identification code, and records the processing progress of the target task item as the initial processing progress.

604. The server divides the task processing request into a plurality of subtask processing requests, and distributes all the subtask processing requests to the executor through the forwarder so that the executor executes any target task item according to the subtask processing requests.

605. After the target task item is executed, the server records that the task processing result of the target task item is completed, and stores the task processing result, the task identification code and the subtask processing request into a buffer.

606. The server side obtains the number of processing tasks corresponding to the task identification codes, and calculates the current processing progress according to the number of processing tasks and the number of target tasks corresponding to the target task items.

607. And the server side sends all current processing progress corresponding to the task identification code to the client side.

Steps 603-607 in this embodiment are similar to steps 503-507 in fig. 5, and detailed description thereof is omitted herein.

608. And the client displays the current processing progress.

609. The client judges whether the value of the current processing progress is larger than the value of the processing progress displayed on the current page of the client, if so, the processing progress displayed on the current page is updated to be the current processing progress, and therefore the current processing progress is displayed.

Steps 608 to 609 in this embodiment are similar to steps 303 to 305 in fig. 3, and detailed descriptions thereof are omitted herein.

610. And the client sends a result query request to the server.

And when the progress of task processing corresponding to the current batch number of the progress bar of the client page reaches 100%, closing websocket connection established with the server, and closing progress bar display. After the client progress bar is displayed and closed, a result query request can be sent to the server.

611. And the server acquires all task processing results of the target task item corresponding to the task identification code according to the result query request.

612. When the current processing progress meets the target processing progress, the server side disconnects the communication connection between the client side and the server side, and clears the task identification code cached by the server side and the current processing progress corresponding to the task identification code.

Steps 611 to 612 in this embodiment are similar to steps 508 to 509 in fig. 5, and detailed descriptions thereof are omitted herein.

According to the task processing method disclosed by the embodiment, when the number of the server clusters is sufficient, approximately 10000 tasks can be processed in 1 minute, and when the batch tasks continue to be increased, the pressure can be shared through the expansion of the clusters. Meanwhile, a large number of tasks are scattered to form individual subtasks, and each subtask is distributed to threads of each server in the cluster for processing. Therefore, the concurrency capacity of large-batch task processing can be improved, and all server resources are reasonably utilized.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

If the scheme involves sensitive information (e.g., user information, business information), it should be noted that the collection, use and handling of the sensitive information requires compliance with laws and regulations of the relevant country and region, and needs to be performed with approval or consent of the corresponding subject (e.g., user or business, etc.).

Referring to fig. 11, fig. 11 is a schematic structural diagram of a task processing system applied to a client according to an embodiment of the present application.

A receiving unit 1101, configured to receive a task processing request initiated by a user at a client, and generate a task identifier corresponding to the task processing request; the task processing request is a processing request generated by a user based on the selected target task item at the client side;

the establishing unit 1102 is configured to establish a communication connection between the client and the server, and send a task identifier, a target task item, and a task processing request to the server, so that the server records a current processing progress of the target task item according to the task identifier; the current processing progress is the processing progress obtained after the server side processes the target task item according to the task processing request;

the receiving unit 1101 is further configured to receive a current processing progress corresponding to the task identifier, and display the current processing progress on the client.

Illustratively, the system further comprises: a judging unit 1103 and an updating unit 1104;

a judging unit 1103, configured to judge whether a value of a current processing progress is greater than a value of a processing progress displayed on a current page of the client;

and an updating unit 1104, configured to update the processing progress displayed on the current page to the current processing progress to display the current processing progress when the value of the current processing progress is greater than the value of the processing progress displayed on the current page of the client.

Illustratively, the system further comprises: closing unit 1105 and query unit 1106;

a closing unit 1105, configured to close the target processing progress displayed on the current page of the client when the current processing progress meets the target processing progress;

the query unit 1106 is configured to query the server for the task processing result of the target task item according to the task identifier, and display the task processing result on the current page of the client.

Referring to fig. 12, fig. 12 is a schematic structural diagram of a task processing system applied to a server according to an embodiment of the present application.

A receiving unit 1201, configured to receive a task identifier, a target task item, and a task processing request that are transmitted by a client through a communication connection; the task processing request is a processing request which is initiated by a user at a client and aims at a target task item, and the task identification code is an identification code corresponding to the task processing request;

An execution unit 1202 for executing the target task item according to the task processing request and obtaining the current processing progress corresponding to the target task item;

and the sending unit 1203 is configured to send all current processing progress corresponding to the task identifier to the client, so that the client displays the current processing progress of the target task item.

Illustratively, the server includes a buffer, and the system further includes: a storage unit 1204;

and a storage unit 1204, configured to store the target task item into the buffer based on the task identification code, and record the processing progress of the target task item as the initial processing progress.

Illustratively, the server includes a repeater, an executor and a buffer, the repeater is used for data transmission, the executor is used for executing the task item, and the system further includes: a dividing unit 1205 and a recording unit 1206;

a dividing unit 1205 for dividing the task processing request into a plurality of subtask processing requests; wherein, any subtask processing request corresponds to any target task item;

the sending unit 1203 is specifically configured to distribute, through the repeater, all the subtask processing requests to the executor, so that the executor executes any target task item according to the subtask processing request;

A recording unit 1206, configured to record that the task processing result of the target task item is completed after the execution of the target task item is completed, and store the task processing result, the task identification code, and the subtask processing request in the buffer; the task processing result, the task identification code and the subtask processing request have an association relation.

Illustratively, the system further comprises: an acquisition unit 1207 and a calculation unit 1208;

an acquisition unit 1207 for acquiring the number of processing tasks corresponding to the task identification code; the number of the processing tasks is the number of the completed task processing results corresponding to the subtask processing requests;

a calculating unit 1208, configured to calculate a current processing progress according to the number of processing tasks and the number of target tasks corresponding to the target task items.

Illustratively, the system further comprises:

the receiving unit 1201 is further configured to receive a result query request sent by the client, and obtain, according to the result query request, all task processing results of the target task item corresponding to the task identifier;

the sending unit 1203 is further configured to send all task processing results to the client.

Illustratively, the server comprises a server, and the system further comprises: a setting-up unit 1209 and a judging unit 1210;

A establishing unit 1209, configured to establish a communication connection between the client and the server;

the storage unit 1204 is specifically configured to store the task identifier to any one of the servers, and determine that any one of the servers is the target server;

an acquiring unit 1207, specifically configured to acquire a current processing progress of the target task item by the target server;

a judging unit 1210, configured to judge whether the identifier of the current processing progress corresponds to the task identifier;

the execution unit 1202 is specifically configured to execute, when the identifier of the current processing progress corresponds to the task identifier, a step of sending all the current processing progress corresponding to the task identifier to the client through the communication connection;

the execution unit 1202 is further configured to, when the identifier of the current processing progress does not correspond to the task identifier, determine that the target server does not exist in the server, and suspend execution of the step of sending all the current processing progress corresponding to the task identifier to the client.

Illustratively, the system further comprises:

the execution unit 1202 is further configured to disconnect the communication connection between the client and the server when the current processing progress meets the target processing progress;

the execution unit 1202 is further configured to clear the task identifier cached by the server and the current processing progress corresponding to the task identifier when the communication connection is disconnected.

Referring to fig. 13, a schematic structural diagram of a task processing device according to an embodiment of the present application includes:

a central processor 1301, a memory 1305, an input/output interface 1304, a wired or wireless network interface 1303, and a power supply 1302;

memory 1305 is a transient memory or persistent memory;

the central processor 1301 is configured to communicate with the memory 1305 and execute instruction operations in the memory 1305 to perform the task processing method in any of the embodiments shown in fig. 2 to 6 described above.

The embodiment of the application also provides a chip system, which is characterized in that the chip system comprises at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used for running a computer program or instructions to execute the task processing method in the embodiment shown in any one of the foregoing fig. 2 to 6.

Those skilled in the art will clearly understand that, for convenience and brevity, the specific working procedures of the above-described systems, apparatuses and units may refer to the corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random access memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Claims

1. A method for processing tasks, applied to a client, the method comprising:

2. The task processing method according to claim 1, wherein after the receiving the current processing progress corresponding to the task identification code, the method further comprises:

3. The task processing method according to claim 1, characterized in that the method further comprises:

4. A method for processing a task, the method being applied to a server, the method comprising:

5. The method for processing tasks according to claim 4, wherein the server includes a buffer, and the method further comprises, after receiving the task identification code, the target task item, and the task processing request transmitted by the client through the communication connection:

6. The method according to claim 4, wherein the server includes a repeater, an executor, and a buffer, the repeater is configured to transfer data, the executor is configured to execute a task item, and the executing the target task item according to the task processing request includes:

7. The task processing method according to claim 6, wherein the obtaining a current processing progress corresponding to the target task item includes:

8. The task processing method according to claim 6, wherein after said transmitting all of the current processing progress corresponding to the task identification code to the client, the method further comprises:

And sending all the task processing results to the client.

9. The method according to claim 4, wherein the server includes a server, and the method further includes, before the receiving the task identification code, the target task item, and the task processing request transmitted by the client through the communication connection:

establishing communication connection between the client and the server;

10. The task processing method according to claim 4, wherein after said transmitting all of the current processing progress corresponding to the task identification code to the client, the method further comprises:

11. A task processing system for application to a client, the system comprising:

12. A task processing system for application to a server, the system comprising:

13. A task processing device, the device comprising:

The memory is a short-term memory or a persistent memory;

the central processor is configured to communicate with the memory and to execute instruction operations in the memory to perform the task processing method of any of claims 1 to 10.

14. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the task processing method of any of claims 1 to 10.