CN113342503A - Real-time progress feedback method, device, equipment and storage medium - Google Patents
Real-time progress feedback method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113342503A CN113342503A CN202110743304.7A CN202110743304A CN113342503A CN 113342503 A CN113342503 A CN 113342503A CN 202110743304 A CN202110743304 A CN 202110743304A CN 113342503 A CN113342503 A CN 113342503A
- Authority
- CN
- China
- Prior art keywords
- processing
- client
- request
- real
- progress
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 258
- 230000005540 biological transmission Effects 0.000 claims abstract description 89
- 238000004891 communication Methods 0.000 claims abstract description 72
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000009795 derivation Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims 1
- 238000011161 development Methods 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000012423 maintenance Methods 0.000 abstract description 2
- 230000002457 bidirectional effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000013524 data verification Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention relates to a development and operation and maintenance technology, and discloses a real-time progress feedback method, which comprises the following steps: the method comprises the steps of establishing long connection communication with a client, deploying a transmission container based on a data processing request sent by the client when the long connection communication is successfully established, constructing a message middleware by using the transmission container, splitting a task according to a request step in the data processing request to obtain a processing subtask, monitoring the execution progress of the processing subtask in real time, and feeding back the processing progress of the processing subtask to the client by using the message middleware. The invention also provides a real-time progress feedback method and device, electronic equipment and a computer readable storage medium. The invention can solve the problem of inaccurate progress feedback of task processing.
Description
Technical Field
The present invention relates to the field of development and operation and maintenance technologies, and in particular, to a real-time progress feedback method and apparatus, an electronic device, and a computer-readable storage medium.
Background
In a system application process of a Browser/Server (Browser/Server) mode, certain data processing tasks which are long in time consumption are often involved, such as data export tasks, and feedback of task progress is often needed to monitor the progress of the data processing tasks in real time.
In the prior art, the commonly used task progress feedback has the following two defects: 1. usually, http short connection polling is used to realize data transmission between two communication parties, but the communication overhead of the short connection polling is relatively high, and if the data processed by the data processing task is too large, the time for further processing the data by the server end may be too long, and if the time for generating a compressed packet according to the derived data is too long, although the server end is still processing the task, the processing time of the server end is too long at this time, which may cause a response to exceed the time limit of the http request, thereby causing the client to disconnect, so that the client cannot obtain the real progress of the data processing task, and only can obtain a message that the data processing task (such as the data deriving task) fails; 2. because the number of databases at the server side is large, when the number of data processing requests is large, data transmission is prone to be disordered, and therefore data transmission is inaccurate and progress feedback of tasks is also inaccurate.
Disclosure of Invention
The invention provides a real-time progress feedback method, a real-time progress feedback device, real-time progress feedback equipment and a storage medium, and mainly aims to solve the problem that progress feedback of task processing is inaccurate.
In order to achieve the above object, the present invention provides a real-time progress feedback method, which includes:
establishing long connection communication with a client;
when the long connection communication is successfully established, deploying a transmission container based on a data processing request sent by the client, and constructing message middleware by using the transmission container;
splitting a task according to the request step in the data processing request to obtain a processing subtask;
monitoring the execution progress of the processing subtasks in real time;
and feeding back the processing progress of the processing subtask to the client by using the message middleware.
Optionally, the establishing long connection communication with the client includes:
acquiring a request handshake containing a protocol to be upgraded sent by a client;
and carrying out protocol conversion according to the protocol to be upgraded, and replying a handshake message to the client after the conversion is successful so as to establish long-connection communication with the client.
Optionally, the deploying a transport container based on the data processing request sent by the client includes:
acquiring a processing instance corresponding to the data request, and constructing a transmission container corresponding to the processing instance;
acquiring the unique identifier of the data processing request and a session object of the data processing request;
constructing a key value pair according to the unique identifier and the session object;
and storing the key value pair into a transmission container corresponding to the processing instance, and determining that the deployment of the transmission container is completed.
Optionally, the constructing message middleware by using the transport container includes:
and summarizing the processing example, the transmission container and the pre-constructed message cluster to obtain the message middleware.
Optionally, the splitting the task according to the request step in the data processing request to obtain a processing subtask includes:
dividing the data processing request according to the request step to obtain a plurality of processing sub-steps;
adding step labels to the plurality of processing substeps, and performing weighted time calculation on the derivation substep to obtain substep time weights;
and generating the processing subtasks based on the substep time weights, and using the step labels as task names of the processing subtasks.
Optionally, the feeding back, by using the message middleware, the processing progress of the processing subtask to the client includes:
utilizing the message middleware to arrange the messages of the processing subtasks to obtain a processing queue;
and feeding back the progress of the processing subtasks in the processing queue to the client by using the message middleware.
Optionally, the using the message middleware to feed back the progress of the processing subtasks in the processing queue to the client includes;
sequentially extracting unique identifications corresponding to the data processing requests of each processing subtask in the processing queue;
judging whether the session object corresponding to the unique identifier is in the transmission container or not;
if the session object corresponding to the unique identifier is not in the transmission container, discarding the processing subtasks, and returning to the step of sequentially extracting the unique identifiers corresponding to the processing subtasks and the data processing request in the processing queue;
and if the session object corresponding to the unique identifier is in the transmission container, feeding back the processing subtask to the client through the processing instance of the transmission container.
In order to solve the above problem, the present invention further provides a real-time progress feedback method and device, where the device includes:
the communication connection module is used for establishing long connection communication with the client;
a message middleware construction module, configured to deploy a transmission container based on the data processing request sent by the client when the long connection communication is successfully established, and construct a message middleware by using the transmission container;
the task splitting module is used for splitting the task according to the request step in the data processing request to obtain a processing subtask;
and the progress feedback module is used for monitoring the execution progress of the processing subtasks in real time and feeding back the processing progress of the processing subtasks to the client by using the message middleware.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the real-time progress feedback method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium having at least one instruction stored therein, where the at least one instruction is executed by a processor in an electronic device to implement the real-time progress feedback method described above.
The invention replaces short connection polling by long connection communication, and compared with a short connection polling mode, a two-way fast channel can be established by only one short connection, thereby reducing communication overhead and improving data transmission speed. Meanwhile, the data processing request is subjected to task splitting according to the steps in the data processing request, processing subtasks of different data processing requests can be further refined, and timeliness of data processing task feedback is guaranteed. And the real-time progress of each processing subtask is fed back by using the transmission container and the message middleware, so that the progress feedback of each processing subtask is more accurate. Therefore, the real-time progress feedback method, the real-time progress feedback device, the electronic equipment and the computer readable storage medium can solve the problem that progress feedback of task processing is inaccurate.
Drawings
Fig. 1 is a schematic flow chart of a real-time progress feedback method according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of a real-time progress feedback device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device for implementing the real-time progress feedback method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a real-time progress feedback method. The execution subject of the real-time progress feedback method includes, but is not limited to, at least one of electronic devices, such as a server and a terminal, which can be configured to execute the method provided by the embodiments of the present application. In other words, the real-time progress feedback method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Fig. 1 is a schematic flow chart of a real-time progress feedback method according to an embodiment of the present invention. In this embodiment, the real-time progress feedback method includes:
and S1, establishing long-connection communication with the client.
In the embodiment of the invention, the client can be an APP, a browser page and the like of the server. Compared with the short connection communication in which two communication parties have data interaction, the long connection communication is established by establishing a connection, and after the data transmission is completed, the connection is disconnected (namely, only one service is transmitted by each connection), and the long connection communication can be kept connected before any communication party is closed.
Specifically, the establishing long connection communication with the client includes:
acquiring a request handshake containing a protocol to be upgraded sent by a client;
and carrying out protocol conversion according to the protocol to be upgraded, and replying a handshake message to the client after the conversion is successful so as to establish long-connection communication with the client.
In the embodiment of the invention, the request handshake can be HTTP request handshake, the protocol to be upgraded can be WebSocket communication protocol, taking client-server communication as an example, the WebSocket communication protocol is a full duplex protocol used in a scene of client-server communication, the HTTP request handshake sent by the client comprises a protocol field to be upgraded (such as Updide: WebSocket) and a protocol Version field (Sec-WebSocket-Version: 13), the client confirms the protocol to be upgraded and the Version of the protocol to be upgraded and sends handshake information to the client, the completion of work of the HTTP protocol is confirmed through the handshake information, and the HTTP protocol is converted into the WebSocket protocol.
In the optional embodiment of the invention, the HTTP short connection is converted into the long connection by using the WebSocket protocol, a bidirectional fast channel can be established only by one-time HTTP request handshake, bidirectional data transmission can be carried out before any one of the client and the server is terminated, and compared with a mode of polling by using the short connection, the communication overhead is reduced, and the data transmission speed is increased.
And S2, when the long connection communication is successfully established, deploying a transmission container based on the data processing request sent by the client, and constructing message middleware by using the transmission container.
In the embodiment of the present invention, the transport container is a service program. For example, in a distributed system, a server has a program for providing a corresponding service at one port, and the program is used for processing a request sent from a client. The data processing request may include: data export request, data storage request, data verification request, and the like.
Specifically, the deploying the transport container based on the data processing request sent by the client includes:
acquiring a processing instance corresponding to the data request, and constructing a transmission container corresponding to the processing instance;
acquiring the unique identifier of the data processing request and a session object of the data processing request;
constructing a key value pair according to the unique identifier and the session object;
and storing the key value pair into a transmission container corresponding to the processing instance, and determining that the deployment of the transmission container is completed.
In an optional embodiment of the present invention, the processing instance refers to a collection of some programs in the server, and has a dynamic memory, and when a data export request is taken as an example, and a file is exported from Oracle, the instance is some database programs capable of supporting the operation of a database. The transmission container can be ConcurrentHashMap, the ConcurrentHashMap is defaulted to divide a hash table into 16 buckets, common operations such as get, put, remove and the like only lock the buckets which need to be used currently, so that only one thread can enter originally, 16 threads can execute at the same time, and concurrency performance is greatly improved.
In the embodiment of the invention, the server constructs the transmission container according to the number of the processing instances, and each instance is used for maintaining one transmission container (ConcurrentHashMap). The unique identification can be uuid, the client generates the unique uuid according to the data processing request, the unique uuid is used as a userId of the websocket communication protocol, and the websocket communication protocol is started. The session object refers to a session, each data processing request has a specific session, and the session is used for storing attributes and configuration information required by sessions of different data processing requests.
In this embodiment of the present invention, the key-value pair may be a key-value pair, where the unique identifier (userId) of the data export request is a value of key, and the session object (session) of the data processing request is a value of value.
Further, the constructing message middleware by using the transport container includes:
and summarizing the processing example, the transmission container and the pre-constructed message cluster to obtain the message middleware.
In this embodiment of the present invention, the message middleware may be Kafka message middleware, where the Kafka message middleware includes a producer node, a consumer node, and a Kafka cluster, and the Kafka cluster stores unique identifiers (userld) of all data processing requests. The producer node is used for distributing the execution progress of each processing instance, and the consumer node is used for feeding back the execution progress of each processing instance.
In an optional embodiment of the present invention, each export instance is used to maintain one transport container, so that distributed high-concurrency data export can be implemented, and data export efficiency is improved.
And S3, splitting the task according to the request step in the data processing request to obtain a processing subtask.
Specifically, the splitting a task according to the request step in the data processing request to obtain a processing subtask includes:
dividing the data processing request according to the request step to obtain a plurality of processing sub-steps;
adding step labels to the plurality of processing substeps, and performing weighted time calculation on the derivation substep to obtain substep time weights;
and generating the processing subtasks based on the substep time weights, and using the step labels as task names of the processing subtasks.
In an alternative embodiment of the present invention, for example, a student achievement derivation request includes A, B, C, D, E five sub-steps, wherein the step labels of the five sub-steps are respectively set as: the method comprises the steps of obtaining a achievement list, inquiring the achievement list, calculating student achievement, updating student achievement and executing completion, wherein the time weight of five substeps of calculating ABCDE is 1:6:1:1:1, wherein the time of the step A is 10%, namely the advancing degree after the step A is completed is increased by 10%, the step B can be determined to need to search 1w pieces of data according to the count inquiry, the step B needs to be inquired 10 times in a paging mode, the step B can be divided into 10 time weight task amounts, the progress is increased by 60%/10%/6% after the execution of each task amount is completed, therefore, the influence on user experience caused by the fact that the progress bar is still for a long time or suddenly increased can be avoided, meanwhile, each export subtask can prompt the task progress to a user according to a step label, and the user experience is further improved.
And S4, monitoring the execution progress of the processing subtasks in real time, and feeding back the processing progress of the processing subtasks to the client by using the message middleware.
Specifically, the feeding back, by using the message middleware, the processing progress of the processing subtask to the client includes:
utilizing the message middleware to arrange the messages of the processing subtasks to obtain a processing queue;
and feeding back the progress of the processing subtasks in the processing queue to the client by using the message middleware.
In the embodiment of the present invention, for example, a producer in each producer node in the message middleware is used to arrange processing subtasks in all processing instances in real time to obtain the processing queue, and based on that a consumer in each consumer node in the message middleware performs real-time processing on the processing subtasks in the queue, whether the processing subtasks are pushed by a current processing instance is determined (determined according to whether a corresponding key value pair can be found in a transmission container).
In detail, the using the message middleware to feed back the progress of the processing subtasks in the processing queue to the client includes;
sequentially extracting unique identifications corresponding to the data processing requests of each processing subtask in the processing queue;
judging whether the session object corresponding to the unique identifier is in the transmission container or not;
if the session object corresponding to the unique identifier is not in the transmission container, discarding the processing subtasks, and returning to the step of sequentially extracting the unique identifiers corresponding to the processing subtasks and the data processing request in the processing queue;
and if the session object corresponding to the unique identifier is in the transmission container, feeding back the processing subtask to the client through the processing instance of the transmission container.
In an alternative embodiment of the present invention, for example, the data export request is an http request when the application system logs in, where the data export request is a websocket request, and subsequent data transmission is actively pushed by the websocket. The application system login request is sent to the instance B, the instance B stores user session information at the moment, and if the websocket request is sent to the instance B when data is exported at the moment, the request can be returned normally; if the websocket requests the instance A, the instance A fails to push the message to the client because the corresponding session cannot be found. Therefore, the problem of multiple instances is solved by using message middleware and transmission containers, one processing instance corresponds to one transmission container, after a consumer of the Kafka message middleware receives a message, a userId is extracted from the message, whether a session corresponding to the userId exists in the transmission container corresponding to the processing instance is judged, if yes, the message is sent to a client through a Websocket, and if not, the message is directly abandoned (because the session is in the transmission containers corresponding to other processing instances), so that the accuracy of real-time data feedback in a distributed system is ensured.
For example, in the dual processing instances a and B, the derived subtasks to be fed back by the processing instance a are: < userId- > { "progress":60} >, wherein "progress":60 means that the task progress is 60%, the kafka producer sends the message to all kafka consumers, the consumers extract the userId from the message after receiving the message, send the userId to the transmission containers corresponding to the instances A and B to judge whether the session corresponding to the userId exists in the instance, if the session is found in the transmission container corresponding to the instance A, the message is sent to the front end { "progress":60} through websocket, and if the session is not found in the transmission container corresponding to the instance B, the message is directly discarded (because the session is fed back through the instance A in the instance A).
The invention replaces short connection polling by long connection communication, and compared with a short connection polling mode, a two-way fast channel can be established by only one short connection, thereby reducing communication overhead and improving data transmission speed. Meanwhile, the data processing request is subjected to task splitting according to the steps in the data processing request, processing subtasks of different data processing requests can be further refined, and timeliness of data processing task feedback is guaranteed. And the real-time progress of each processing subtask is fed back by using the transmission container and the message middleware, so that the progress feedback of each processing subtask is more accurate. Therefore, the real-time progress feedback method provided by the invention can solve the problem that the progress feedback of task processing is inaccurate.
Fig. 2 is a functional block diagram of a real-time progress feedback apparatus according to an embodiment of the present invention.
The real-time progress feedback device 100 of the present invention may be installed in an electronic device. According to the implemented functions, the real-time progress feedback device 100 may include a communication connection module 101, a message middleware construction module 102, a task splitting module 103, and a progress feedback module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the communication connection module 101 is configured to establish long connection communication with a client.
In the embodiment of the invention, the client can be an APP, a browser page and the like of the server. Compared with the short connection communication in which two communication parties have data interaction, the long connection communication is established by establishing a connection, and after the data transmission is completed, the connection is disconnected (namely, only one service is transmitted by each connection), and the long connection communication can be kept connected before any communication party is closed.
Specifically, the communication connection module 101 establishes long connection communication with a client by performing the following operations:
acquiring a request handshake containing a protocol to be upgraded sent by a client;
and carrying out protocol conversion according to the protocol to be upgraded, and replying a handshake message to the client after the conversion is successful so as to establish long-connection communication with the client.
In the embodiment of the invention, the request handshake can be HTTP request handshake, the protocol to be upgraded can be WebSocket communication protocol, taking client-server communication as an example, the WebSocket communication protocol is a full duplex protocol used in a scene of client-server communication, the HTTP request handshake sent by the client comprises a protocol field to be upgraded (such as Updide: WebSocket) and a protocol Version field (Sec-WebSocket-Version: 13), the client confirms the protocol to be upgraded and the Version of the protocol to be upgraded and sends handshake information to the client, the completion of work of the HTTP protocol is confirmed through the handshake information, and the HTTP protocol is converted into the WebSocket protocol.
In the optional embodiment of the invention, the HTTP short connection is converted into the long connection by using the WebSocket protocol, a bidirectional fast channel can be established only by one-time HTTP request handshake, bidirectional data transmission can be carried out before any one of the client and the server is terminated, and compared with a mode of polling by using the short connection, the communication overhead is reduced, and the data transmission speed is increased.
The message middleware building module 102 is configured to deploy a transmission container based on the data processing request sent by the client when the long connection communication is successfully established, and build a message middleware by using the transmission container.
In the embodiment of the present invention, the transport container is a service program, for example, in a distributed system, a port of a server has a program providing a corresponding service, and the program processes a request sent from a client. The data processing request may include: data export request, data storage request, data verification request, and the like.
Specifically, the message middleware building module 102 deploys the transport container by performing the following operations:
acquiring a processing instance corresponding to the data request, and constructing a transmission container corresponding to the processing instance;
acquiring the unique identifier of the data processing request and a session object of the data processing request;
constructing a key value pair according to the unique identifier and the session object;
and storing the key value pair into a transmission container corresponding to the processing instance, and determining that the deployment of the transmission container is completed.
In an optional embodiment of the present invention, the processing instance refers to a collection of some programs in the server, and has a dynamic memory, and when a data export request is taken as an example, and a file is exported from Oracle, the instance is some database programs capable of supporting the operation of a database. The transmission container can be ConcurrentHashMap, the ConcurrentHashMap is defaulted to divide a hash table into 16 buckets, common operations such as get, put, remove and the like only lock the buckets which need to be used currently, so that only one thread can enter originally, 16 threads can execute at the same time, and concurrency performance is greatly improved.
In the embodiment of the invention, the server constructs the transmission container according to the number of the processing instances, and each instance is used for maintaining one transmission container (ConcurrentHashMap). The unique identification can be uuid, the client generates the unique uuid according to the data processing request, the unique uuid is used as a userId of the websocket communication protocol, and the websocket communication protocol is started. The session object refers to a session, each data processing request has a specific session, and the session is used for storing attributes and configuration information required by sessions of different data processing requests.
In this embodiment of the present invention, the key-value pair may be a key-value pair, where the unique identifier (userId) of the data export request is a value of key, and the session object (session) of the data processing request is a value of value.
Further, the message middleware building module 102 builds message middleware by performing the following operations:
and summarizing the processing example, the transmission container and the pre-constructed message cluster to obtain the message middleware.
In this embodiment of the present invention, the message middleware may be Kafka message middleware, where the Kafka message middleware includes a producer node, a consumer node, and a Kafka cluster, and the Kafka cluster stores unique identifiers (userld) of all data processing requests. The producer node is used for distributing the execution progress of each processing instance, and the consumer node is used for feeding back the execution progress of each processing instance.
In an optional embodiment of the present invention, each export instance is used to maintain one transport container, so that distributed high-concurrency data export can be implemented, and data export efficiency is improved.
The task splitting module 103 is configured to split a task according to the request step in the data processing request to obtain a processing subtask.
Specifically, the task splitting module 103 obtains a processing subtask by performing the following operations:
dividing the data processing request according to the request step to obtain a plurality of processing sub-steps;
adding step labels to the plurality of processing substeps, and performing weighted time calculation on the derivation substep to obtain substep time weights;
and generating the processing subtasks based on the substep time weights, and using the step labels as task names of the processing subtasks.
In an alternative embodiment of the present invention, for example, a student achievement derivation request includes A, B, C, D, E five sub-steps, wherein the step labels of the five sub-steps are respectively set as: the method comprises the steps of obtaining a achievement list, inquiring the achievement list, calculating student achievement, updating student achievement and executing completion, wherein the time weight of five substeps of calculating ABCDE is 1:6:1:1:1, wherein the time of the step A is 10%, namely the advancing degree after the step A is completed is increased by 10%, the step B can be determined to need to search 1w pieces of data according to the count inquiry, the step B needs to be inquired 10 times in a paging mode, the step B can be divided into 10 time weight task amounts, the progress is increased by 60%/10%/6% after the execution of each task amount is completed, therefore, the influence on user experience caused by the fact that the progress bar is still for a long time or suddenly increased can be avoided, meanwhile, each export subtask can prompt the task progress to a user according to a step label, and the user experience is further improved.
The progress feedback module 104 is configured to monitor the execution progress of the processing subtask in real time, and feed back the processing progress of the processing subtask to the client by using the message middleware.
Specifically, the progress feedback module 104 feeds back the processing progress of the processing subtask to the client by performing the following operations:
utilizing the message middleware to arrange the messages of the processing subtasks to obtain a processing queue;
and feeding back the progress of the processing subtasks in the processing queue to the client by using the message middleware.
In the embodiment of the present invention, for example, a producer in each producer node in the message middleware is used to arrange processing subtasks in all processing instances in real time to obtain the processing queue, and based on that a consumer in each consumer node in the message middleware performs real-time processing on the processing subtasks in the queue, whether the processing subtasks are pushed by a current processing instance is determined (determined according to whether a corresponding key value pair can be found in a transmission container).
In detail, the progress feedback module 104 feeds back the progress of the processing subtasks in the processing queue to the client using the message middleware by performing the following operations:
sequentially extracting unique identifications corresponding to the data processing requests of each processing subtask in the processing queue;
judging whether the session object corresponding to the unique identifier is in the transmission container or not;
if the session object corresponding to the unique identifier is not in the transmission container, discarding the processing subtasks, and returning to the step of sequentially extracting the unique identifiers corresponding to the processing subtasks and the data processing request in the processing queue;
and if the session object corresponding to the unique identifier is in the transmission container, feeding back the processing subtask to the client through the processing instance of the transmission container.
In an alternative embodiment of the present invention, for example, the data export request is an http request when the application system logs in, where the data export request is a websocket request, and subsequent data transmission is actively pushed by the websocket. The application system login request is sent to the instance B, the instance B stores user session information at the moment, and if the websocket request is sent to the instance B when data is exported at the moment, the request can be returned normally; if the websocket requests the instance A, the instance A fails to push the message to the client because the corresponding session cannot be found. Therefore, the problem of multiple instances is solved by using message middleware and transmission containers, one processing instance corresponds to one transmission container, after a consumer of the Kafka message middleware receives a message, a userId is extracted from the message, whether a session corresponding to the userId exists in the transmission container corresponding to the processing instance is judged, if yes, the message is sent to a client through a Websocket, and if not, the message is directly abandoned (because the session is in the transmission containers corresponding to other processing instances), so that the accuracy of real-time data feedback in a distributed system is ensured.
For example, in the dual processing instances a and B, the derived subtasks to be fed back by the processing instance a are: < userId- > { "progress":60} >, wherein "progress":60 means that the task progress is 60%, the kafka producer sends the message to all kafka consumers, the consumers extract the userId from the message after receiving the message, send the userId to the transmission containers corresponding to the instances A and B to judge whether the session corresponding to the userId exists in the instance, if the session is found in the transmission container corresponding to the instance A, the message is sent to the front end { "progress":60} through websocket, and if the session is not found in the transmission container corresponding to the instance B, the message is directly discarded (because the session is fed back through the instance A in the instance A).
The invention replaces short connection polling by long connection communication, and compared with a short connection polling mode, a two-way fast channel can be established by only one short connection, thereby reducing communication overhead and improving data transmission speed. Meanwhile, the data processing request is subjected to task splitting according to the steps in the data processing request, processing subtasks of different data processing requests can be further refined, and timeliness of data processing task feedback is guaranteed. And the real-time progress of each processing subtask is fed back by using the transmission container and the message middleware, so that the progress feedback of each processing subtask is more accurate. Therefore, the real-time progress feedback device provided by the invention can solve the problem that the progress feedback of task processing is inaccurate.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a real-time progress feedback method according to an embodiment of the present invention, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, where the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114,
a memory 113 for storing a computer program such as a real-time progress feedback method program;
in an embodiment of the present application, when the processor 111 is configured to execute the program stored in the memory 113, the method for real-time progress feedback provided in any one of the foregoing method embodiments is implemented, and includes:
establishing long connection communication with a client;
when the long connection communication is successfully established, deploying a transmission container based on a data processing request sent by the client, and constructing message middleware by using the transmission container;
splitting a task according to the request step in the data processing request to obtain a processing subtask;
monitoring the execution progress of the processing subtasks in real time;
and feeding back the processing progress of the processing subtask to the client by using the message middleware.
The communication bus 114 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 114 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface 112 is used for communication between the above-described electronic apparatus and other apparatuses.
The memory 113 may include a Random Access Memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 113 may also be at least one storage device located remotely from the processor 111.
The processor 111 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the integrated circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
establishing long connection communication with a client;
when the long connection communication is successfully established, deploying a transmission container based on a data processing request sent by the client, and constructing message middleware by using the transmission container;
splitting a task according to the request step in the data processing request to obtain a processing subtask;
monitoring the execution progress of the processing subtasks in real time;
and feeding back the processing progress of the processing subtask to the client by using the message middleware.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A real-time progress feedback method, the method comprising:
establishing long connection communication with a client;
when the long connection communication is successfully established, deploying a transmission container based on a data processing request sent by the client, and constructing message middleware by using the transmission container;
splitting a task according to the request step in the data processing request to obtain a processing subtask;
monitoring the execution progress of the processing subtasks in real time;
and feeding back the processing progress of the processing subtask to the client by using the message middleware.
2. The real-time progress feedback method of claim 1, wherein establishing long connection communication with a client comprises:
acquiring a request handshake containing a protocol to be upgraded sent by a client;
and carrying out protocol conversion according to the protocol to be upgraded, and replying a handshake message to the client after the conversion is successful so as to establish long-connection communication with the client.
3. The real-time progress feedback method of claim 1, wherein deploying a transport container based on a data processing request sent by the client comprises:
acquiring a processing instance corresponding to the data request, and constructing a transmission container corresponding to the processing instance;
acquiring the unique identifier of the data processing request and a session object of the data processing request;
constructing a key value pair according to the unique identifier and the session object;
and storing the key value pair into a transmission container corresponding to the processing instance, and determining that the deployment of the transmission container is completed.
4. The real-time progress feedback method of claim 3, wherein the constructing message middleware using the transport container includes:
and summarizing the processing example, the transmission container and the pre-constructed message cluster to obtain the message middleware.
5. The real-time progress feedback method according to any one of claims 1 to 3, wherein the splitting of the task according to the request step in the data processing request to obtain the processing subtask comprises:
dividing the data processing request according to the request step to obtain a plurality of processing sub-steps;
adding step labels to the plurality of processing substeps, and performing weighted time calculation on the derivation substep to obtain substep time weights;
and generating the processing subtasks based on the substep time weights, and using the step labels as task names of the processing subtasks.
6. The real-time progress feedback method according to any one of claims 1 to 3, wherein the feeding back the processing progress of the processing subtask to the client by using the message middleware comprises:
utilizing the message middleware to arrange the messages of the processing subtasks to obtain a processing queue;
and feeding back the progress of the processing subtasks in the processing queue to the client by using the message middleware.
7. The real-time progress feedback method of claim 6, wherein the feeding back, with the message middleware, the progress of the processing subtasks in the processing queue to the client comprises;
sequentially extracting unique identifications corresponding to the data processing requests of each processing subtask in the processing queue;
judging whether the session object corresponding to the unique identifier is in the transmission container or not;
if the session object corresponding to the unique identifier is not in the transmission container, discarding the processing subtasks, and returning to the step of sequentially extracting the unique identifiers corresponding to the processing subtasks and the data processing request in the processing queue;
and if the session object corresponding to the unique identifier is in the transmission container, feeding back the processing subtask to the client through the processing instance of the transmission container.
8. A real-time progress feedback device, the device comprising:
the communication connection module is used for establishing long connection communication with the client;
a message middleware construction module, configured to deploy a transmission container based on the data processing request sent by the client when the long connection communication is successfully established, and construct a message middleware by using the transmission container;
the task splitting module is used for splitting the task according to the request step in the data processing request to obtain a processing subtask;
and the progress feedback module is used for monitoring the execution progress of the processing subtasks in real time and feeding back the processing progress of the processing subtasks to the client by using the message middleware.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a real-time progress feedback method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the real-time progress feedback method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110743304.7A CN113342503B (en) | 2021-06-30 | 2021-06-30 | Real-time progress feedback method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110743304.7A CN113342503B (en) | 2021-06-30 | 2021-06-30 | Real-time progress feedback method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113342503A true CN113342503A (en) | 2021-09-03 |
| CN113342503B CN113342503B (en) | 2024-02-13 |
Family
ID=77482078
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110743304.7A Active CN113342503B (en) | 2021-06-30 | 2021-06-30 | Real-time progress feedback method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342503B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118708A (en) * | 2022-08-25 | 2022-09-27 | 飞狐信息技术（天津）有限公司 | HTTP proxy method and device for message middleware |
| CN115333713A (en) * | 2022-10-14 | 2022-11-11 | 海南浙江大学研究院 | Full-duplex underwater acoustic communication medium access control method based on mutual interference avoidance |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105407180A (en) * | 2015-12-29 | 2016-03-16 | 曙光信息产业(北京)有限公司 | Server message pushing method and device |
| CN110704164A (en) * | 2019-09-30 | 2020-01-17 | 珠海市新德汇信息技术有限公司 | Cloud native application platform construction method based on Kubernetes technology |
| CN110753129A (en) * | 2019-10-29 | 2020-02-04 | 深圳前海微众银行股份有限公司 | Message transmission method, system, device, equipment and computer readable storage medium |
| CN112202879A (en) * | 2020-09-29 | 2021-01-08 | 中移（杭州）信息技术有限公司 | Middleware management method and device, electronic equipment and storage medium |
-
2021
- 2021-06-30 CN CN202110743304.7A patent/CN113342503B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105407180A (en) * | 2015-12-29 | 2016-03-16 | 曙光信息产业(北京)有限公司 | Server message pushing method and device |
| CN110704164A (en) * | 2019-09-30 | 2020-01-17 | 珠海市新德汇信息技术有限公司 | Cloud native application platform construction method based on Kubernetes technology |
| CN110753129A (en) * | 2019-10-29 | 2020-02-04 | 深圳前海微众银行股份有限公司 | Message transmission method, system, device, equipment and computer readable storage medium |
| CN112202879A (en) * | 2020-09-29 | 2021-01-08 | 中移（杭州）信息技术有限公司 | Middleware management method and device, electronic equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118708A (en) * | 2022-08-25 | 2022-09-27 | 飞狐信息技术（天津）有限公司 | HTTP proxy method and device for message middleware |
| CN115333713A (en) * | 2022-10-14 | 2022-11-11 | 海南浙江大学研究院 | Full-duplex underwater acoustic communication medium access control method based on mutual interference avoidance |
| CN115333713B (en) * | 2022-10-14 | 2023-02-07 | 海南浙江大学研究院 | Full-duplex underwater acoustic communication medium access control method based on mutual interference avoidance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113342503B (en) | 2024-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110113381B (en) | Method and device for subscribing theme in block chain | |
| CN109600375B (en) | Message tracking method and device, electronic equipment and storage medium | |
| CN112988485B (en) | Simulation test method and device for electric power Internet of things equipment | |
| CN114024972B (en) | Long connection communication method, system, device, equipment and storage medium | |
| CN113342503A (en) | Real-time progress feedback method, device, equipment and storage medium | |
| CN112256523B (en) | Service data processing method and device | |
| CN109254854A (en) | Asynchronous invoking method, computer installation and storage medium | |
| CN113064735A (en) | Service processing method and device | |
| CN104468248A (en) | Service performance monitoring method, reverse proxy server, statistical analysis server and system | |
| CN115951923B (en) | Subscription event management method, display system, device and storage medium | |
| CN111738721A (en) | Block chain transaction monitoring method and related device | |
| CN112838980A (en) | Message processing method, system, device, electronic equipment and storage medium | |
| CN114490100A (en) | Message queue telemetry transmission load balancing method and device and server | |
| CN113965628A (en) | Message scheduling method, server and storage medium | |
| CN114050909B (en) | Exercise method, system and electronic equipment for simulating mail | |
| CN114356713A (en) | Thread pool monitoring method and device, electronic equipment and storage medium | |
| CN110474954B (en) | Block chain-based file sharing method and device, computer equipment and readable storage medium | |
| CN113392343A (en) | Data extraction method, device, medium and computer program product | |
| CN110569673A (en) | Data file processing method, device, equipment and storage medium | |
| CN113254097A (en) | Configuration information issuing method and device, electronic equipment and storage medium | |
| CN113381907A (en) | Log collection method and device, electronic equipment and storage medium | |
| CN112333262A (en) | Data updating prompting method and device, computer equipment and readable storage medium | |
| CN112988806A (en) | Data processing method and device | |
| CN114938367B (en) | Batch job processing method, system, equipment and medium | |
| CN117527519A (en) | Message push link tracking method, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |