CN113176967B - Data processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113176967B CN113176967B CN202110475234.1A CN202110475234A CN113176967B CN 113176967 B CN113176967 B CN 113176967B CN 202110475234 A CN202110475234 A CN 202110475234A CN 113176967 B CN113176967 B CN 113176967B
- Authority
- CN
- China
- Prior art keywords
- message
- data
- server
- processing operation
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the application provides a data processing method and device, electronic equipment and a computer readable storage medium; the method comprises the following steps: under the condition that the execution of the first processing operation corresponding to the first message fails, the first server writes first data corresponding to the first message into a retry queue; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; the first server starts the first thread and re-performs the first processing operation by trying the first data in the queue through the first thread Cheng Duichong; the first thread is different from the second thread based on the second thread performing a plurality of processing operations. The application can further cause the thread to re-execute the operation with the execution failure, so as to ensure that the operation with the execution failure is continuously executed downwards from the beginning, and ensure the stability of the execution of the message.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and apparatus for processing data, an electronic device, and a computer readable storage medium.
Background
With popularization of the Internet, the requirements of partial industries on high concurrency and quick response of the service end are higher and higher; such as very high flows in a short period of time that may occur with some systems. For conventional systems, such extremely high traffic in a short period of time is catastrophic, may directly crash the server,
in this regard, in the prior art, a Message Queue (MQ) is introduced, where the Message Queue is a communication method between application programs, and is generally used for transferring data between two services in different application programs, and has functions of traffic clipping, decoupling and asynchronization by establishing a Message persistent storage manner between two or more services. For example, for A, B two services with data transfer requirements (service a needs to transfer data to service B), since a and B have MQ services, MQ servers are added to the system where a and B are located; wherein A is used as a message provider and is responsible for producing a message; the MQ server is used to store messages and process some of the message sending or receiving logic; b is responsible for consuming messages as a message consumer.
It can be seen that the introduction of MQ alleviates the problems of high concurrency and strong coupling of the server. However, in the process of post-processing the message received by the message consumer, if one of the processing links fails, for example, the writing operation fails, since the previous step is already executed, the data of the current state cannot be saved, that is, the processing can only be executed downwards from the step after the step of executing the failure, thereby reducing the processing stability of the message at the consumer side.
Disclosure of Invention
An embodiment of the application aims to provide a data processing method and device, electronic equipment and a computer readable storage medium, so as to ensure the stability of message execution. The specific technical scheme is as follows:
in a first aspect of the present application, there is provided a data processing method, including: under the condition that the execution of a first processing operation corresponding to a first message fails, the first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to the first server consuming the message by a second server producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue by the first thread, wherein the plurality of processing operations are executed based on a second thread, the first thread being different from the second thread.
In a second aspect of the present application, there is also provided a data processing apparatus, applied to a first server, including: the first writing module is used for writing first data corresponding to a first message into a retry queue under the condition that the execution of a first processing operation corresponding to the first message fails, wherein the first message is sent to the first server for consuming the message by a second server for producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; and the first processing module is used for starting a first thread and re-executing the first processing operation on the first data in the retry queue through the first thread, wherein the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
In yet another aspect of the present application, there is also provided a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform the method of processing data as described in any of the above.
In yet another aspect of the application there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method of processing data as described in any of the above.
In the data processing method provided by the embodiment of the application, under the condition that the execution of the first processing operation corresponding to the first message fails, the first data corresponding to the first message is written into the retry queue, and then the first thread is started to execute the write operation again on the first data; because the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing of the first message by the first server, that is, the first server receiving the message can re-execute the processing operation on the data with the execution failure by the thread after the execution failure of the message execution process, that is, the re-execution of the execution failure operation by the thread after the thread is executed again, so as to ensure that the execution is continued downwards from the beginning of the execution failure operation, thereby avoiding the problem that the whole process corresponding to the incomplete execution message is not completed because the first server needs to execute downwards from the step with the execution failure after the execution failure occurs. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be re-executed on the data of the execution failure in the retry queue by the thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flowchart of a method for processing data according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a data retry process according to an embodiment of the present application;
FIG. 3 is a second flowchart of a method for processing data according to an embodiment of the present application.
FIG. 4 is a third flowchart of a method for processing data according to an embodiment of the present application;
FIG. 5 is a flowchart of a method for processing data according to an embodiment of the present application;
FIG. 6 is a second schematic diagram of a data retry process according to an embodiment of the present application;
FIG. 7 is a flowchart showing a method of processing data according to an embodiment of the present application;
FIG. 8 is a flowchart showing a method of processing data according to an embodiment of the present application;
FIG. 9 is a third diagram illustrating a data retry process according to an embodiment of the application;
FIG. 10 is a diagram illustrating statistics and reporting of abnormal data according to an embodiment of the present application;
FIG. 11 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
The embodiment of the application provides a data processing method, as shown in fig. 1, comprising the following steps:
step 102, under the condition that the execution of a first processing operation corresponding to a first message fails, the first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to a first server consuming the message by a second server producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
in step 104, the first server starts a first thread and re-executes the first processing operation via the first data in the first line Cheng Duichong queue, wherein the first thread is different from the second thread based on the second thread performing the plurality of processing operations.
Through the steps 102 and 104, in the case that the execution of the first processing operation corresponding to the first message fails, writing the first data corresponding to the first message into the retry queue, and then restarting the first thread to execute the write operation again on the first data; because the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing of the first message by the first server, that is, the first server receiving the message can re-execute the processing operation on the data with the execution failure by the thread after the execution failure of the message execution process, that is, the re-execution of the execution failure operation by the thread after the thread is executed again, so as to ensure that the execution is continued downwards from the beginning of the execution failure operation, thereby avoiding the problem that the whole process corresponding to the incomplete execution message is not completed because the first server needs to execute downwards from the step with the execution failure after the execution failure occurs. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be re-executed on the data of the execution failure in the retry queue by the thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
The above steps 102 to 104 are explained below in conjunction with a specific application scenario.
As shown in fig. 2, after the producer service a in the second server is online, a message is sent to the MQ server, and then the MQ server obtains the message by pushing or actively pulling by the consumer service B in the first server, where the message goes through a plurality of processing operations after entering the first server, for example: status update, binning, writing logs (corresponding to a plurality of processing operations in an embodiment of the application that process the first message). When one of the processing operations fails to perform, for example, a binning operation (corresponding to the first processing operation), since the previous step (state update) has already been performed, the rollback cannot be performed, and only the data of the performed operation of the state update can be saved, that is, only the processing operation (write log) after the current execution failure can be performed downwards, and the plurality of processing operations cannot be completely performed, which results in lower stability of processing of the message at the consumer side. In this case, the data corresponding to the to-be-binned may be written into a retry queue of the first server, where the retry queue is used to store the data that fails in binning operation, and a thread (corresponding to the first thread in the embodiment of the present application, a thread that normally executes the plurality of processing operations is a second thread) is started in the first server, so as to execute binning operation again on the data corresponding to the failed binning operation, so that the first server may continue to execute binning operation again based on the data fetched from the retry queue, and then execute the steps after binning.
In an optional implementation manner of the embodiment of the present application, for the first server involved in step 102 to write the first data corresponding to the first message into the retry queue, the method further may include: the first server writes the first data and a timestamp corresponding to the first data into a retry queue, wherein the timestamp is execution time of the first processing operation.
It can be seen that, in the embodiment of the present application, in addition to writing the first data corresponding to the first processing operation that fails to be executed into the retry queue, the execution time (timestamp) of the first processing operation corresponding to the first data needs to be written into the retry queue. After entering the first server with the first message, a plurality of processing operations are performed, for example: status update, binning, log writing are examples. The time stamps of the three processing operations are respectively a first time, a second time and a third time, and the first time is earlier than the second time and the second time is earlier than the third time. When a certain processing operation fails to be executed, for example, when the execution of the warehousing operation fails, the first server needs to write the data corresponding to the warehousing operation into the retry queue at the second time.
In another alternative implementation manner of the embodiment of the present application, as shown in fig. 3, the method of the embodiment of the present application may further include, in addition to the method steps in fig. 1, the following steps:
step 302, before the first server re-executes the first processing operation on the first data through the first thread, and in the case that the first data also corresponds to the second message, the first server determines an execution state of the second processing operation corresponding to the second message; wherein the execution result of the second processing operation will overlap the execution result of the first processing operation; the second processing operation is performed at a time later than the first processing operation,
step 304, under the condition that the execution of the second processing operation fails, the first server writes the first data corresponding to the second message into a retry queue;
in step 306, the first server prohibits starting the first thread if the second processing operation is successful and the first processing operation has not been re-performed on the first data by the first thread.
As can be seen from steps 302 to 306, in the case that there may be a plurality of corresponding messages (the first data corresponds to the first message and the second message) in the same data, for example, the first data is 1, and the first message and the second message may update the first data. If the processing operation of the first message is to update 1 to 2, the processing operation of the second message is to update data to 3 if the first data is 1 or to update data to 3 if the first data is 2, and the execution time of the first message is earlier than that of the second message. That is, the execution result of the processing operation corresponding to the second message will cover the execution result of the processing operation corresponding to the first message, that is, when the first message and the second message are normally executed, 1 will be updated to 3. However, in the case that the execution of the first message fails, the corresponding data is written into the retry queue, in this case, it is necessary to further determine whether the second message has been executed, and if the second message has been executed, the data corresponding to the first message written into the retry queue due to the execution failure does not need to be executed again. That is, the processing procedure of the normal service is that "1" is updated to "2" and then "2" is updated to "3", if the execution of the operation for the update in the current first message fails, after writing the corresponding data thereof in the retry queue, it is necessary to further determine whether the second message has been successfully executed, that is, whether the number "1" has been updated to "3", if it has been updated to "3", it is unnecessary to start a new thread to execute the write operation on the data in the retry queue, and if it has not been updated to "3", it is necessary to execute the write operation again on the data corresponding to the first message in the retry queue.
In another alternative implementation manner of the embodiment of the present application, as shown in fig. 4, the method of the embodiment of the present application may further include, in addition to the method steps in fig. 1:
step 402, performing persistence processing on data corresponding to the message in the retransmission queue under the condition that at least one of the following is satisfied: before the service corresponding to the message in the retry queue stops, the number of the messages in the retry queue is greater than a preset threshold.
Note that before the processing of the message is stopped, it means before all of the processing operations corresponding to the message have been completed, because the corresponding data is easily lost after the processing of the processing operations is completed. In addition, in order to reduce the burden of data storage in the alternative implementation manner of the embodiment of the present application, the data in the retry queue in the embodiment of the present application is preferably memory data, that is, corresponding data may be lost when a plurality of processing operations are completed. In addition, if too much data is stored in the retry queue, memory overflow, i.e., data loss, may also result. If the data is lost, the write operation cannot be re-performed on the data in the re-queue. Therefore, in the embodiment of the present application, before the processing of the message is stopped, or when the data amount of the data in the retry queue is greater than or equal to the preset threshold, the data in the retry queue needs to be moved to the memory medium, that is, the persistence of the data. Only one mode is needed in the specific application scene of the embodiment of the application, the two modes can be combined for use, and the specific mode can be correspondingly set in combination with specific actual conditions.
It should be noted that, in the embodiment of the present application, the memory medium may be selected as a disk of the server, and in other application scenarios, may also be a proprietary storage unit, such as MySQL in a relational database management system; the performance requirements, if any, may also be stored in a distributed cache, such as a remote dictionary server (REmote DIctionary Server, redis).
When the persistence operation is triggered, the message processing problem is severe, reporting abnormality can be notified at the moment, and specific notification level can be set correspondingly according to actual requirements. As shown in fig. 5, in addition to the method steps in fig. 4, after performing persistence processing on data corresponding to the message in the retry queue, the method in the embodiment of the present application may further include:
step 502, the first server reports a first notification message to a third server, where the first notification message is used to indicate that the write operation is abnormal, and the third server is a server in the alarm platform.
It can be seen that if more and more data is present in the current retry queue, this indicates that the current processing load through the retry queue is heavier and more, a notification message needs to be sent to alert the relevant personnel. As shown in fig. 6, when the target thread processes the data in the retry queue, the target thread performs persistence and performs an exception report (reports the first notification message).
In yet another alternative implementation manner of the embodiment of the present application, as shown in fig. 7, the method steps of the embodiment of the present application may further include, in addition to the method steps in fig. 1:
in step 702, the first server deletes the first data from the retry queue if the first processing operation is successful by the first thread to re-execute the first processing operation on the first data.
As can be seen from the above step 702, for the data in the retry queue for which the write operation has been successfully performed, the data can be deleted from the retry queue without re-performing the write operation, so as to make room for storing other data, thereby avoiding redundancy of the data in the retry queue.
In another optional implementation manner of the embodiment of the present application, as shown in fig. 8, in the case that the execution of the first write operation corresponding to the first message fails, before the first server involved in step 102 of the embodiment of the present application writes the first data corresponding to the first message into the retry queue, the method steps of the embodiment of the present application may further include:
step 802, the first server re-executes the first processing operation;
step 804, the first server writes the first data into the log and stores the log to the disk in case that the re-execution of the first processing operation fails;
step 806, the first server counts the abnormal attributes of the log in the disk, and reports a second notification message to the third server according to the statistical result, where the second notification message is used to notify that the write operation is abnormal.
As can be seen from the above step 802, in the case that the first processing operation fails to be executed, the write operation can be directly re-executed on the first processing operation, and if the first processing operation can be directly re-executed successfully, the first data corresponding to the first processing operation does not need to be written into the retry queue. It should be noted that, in the embodiment of the present application, the processing operation may be re-performed multiple times, and the number of times of re-performing the processing operation may be preset, for example, 3 times, 5 times, etc. For this step 802 in a specific application scenario, as shown in fig. 9, after the producer service a in the second server is online, a message is sent to the MQ server, and then the MQ server obtains the message by pushing or actively pulling by the consumer service B in the first server, where after the message enters the first server, a plurality of processing operations are performed, for example: status update, binning, writing logs (corresponding to a plurality of processing operations in an embodiment of the application that process the first message). When one of the processing operations fails to be executed, for example, a warehousing operation (corresponding to a first processing operation), the warehousing operation is retried after the warehousing operation fails, the upper limit of the retrying is three times, if the retrying is successful within three times, the data corresponding to the warehousing is not required to be written into a retry queue, and if the retrying is still not successfully executed after three times, the data corresponding to the warehousing is written into the retry queue.
In addition, in a specific application scenario, the manner referred to in the above steps 804 and 806 may be that, as shown in fig. 10, the corresponding modules (for example, the gathering delivery module) are started at the same time when the service is started. When the warehouse-in fails, firstly, the data is written into a log, the log is stored in a disk, and different types (such as the warehouse-in failure, the full queue data and the like) can be defined in the log for later data statistics. After the abnormal logs are written into the disk, the collection and delivery module analyzes the saved logs and delivers analysis results to the statistics center; the statistics center aggregates the log data such as counting abnormal attributes, e.g., abnormal types, abnormal numbers, storing statistics, etc. After the statistics is completed, an alarm center is called to send an alarm short message or mail (notification message), and meanwhile, the statistics result can be sent to a display module to display abnormal conditions in different dimensions.
As shown in fig. 11, the embodiment of the present application further provides a data processing apparatus, where the apparatus includes:
a first writing module 112, configured to write, in a retry queue, first data corresponding to a first message when execution of a first processing operation corresponding to the first message fails, where the first message is sent to a first server that consumes a message by a second server that produces the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
the first processing module 114 is configured to start a first thread and re-execute a first processing operation on the first data in the queue via a first thread Cheng Duichong, wherein the first thread is different from the second thread based on the second thread performing a plurality of processing operations.
Through the device in the embodiment of the application, under the condition that the execution of the first processing operation corresponding to the first message fails, the first data corresponding to the first message is written into the retry queue, and then the first thread is started to execute the write operation again on the first data; because the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing of the first message by the first server, that is, the first server receiving the message can re-execute the processing operation on the data with the execution failure by the thread after the execution failure of the message execution process, that is, the re-execution of the execution failure operation by the thread after the thread is executed again, so as to ensure that the execution is continued downwards from the beginning of the execution failure operation, thereby avoiding the problem that the whole process corresponding to the incomplete execution message is not completed because the first server needs to execute downwards from the step with the execution failure after the execution failure occurs. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be re-executed on the data of the execution failure in the retry queue by the thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
Optionally, the first writing module 112 in the embodiment of the present application may further include: and the writing unit is used for writing the first data and a timestamp corresponding to the first data into the retry queue, wherein the timestamp is the execution time of the first processing operation.
Optionally, the device in the embodiment of the present application may further include: a determining module, configured to determine, before the first processing operation is re-performed on the first data by the first thread, and in a case where the first data also corresponds to the second message, an execution state of the second processing operation corresponding to the second message; wherein the execution result of the second processing operation will overlap the execution result of the first processing operation; the execution time of the second processing operation is later than the execution time of the first processing operation; the second writing module is used for writing the first data corresponding to the second message into the retry queue under the condition that the execution of the second processing operation fails; and the prohibiting module is used for prohibiting starting of the first thread under the condition that the second processing operation is successful and the first processing operation is not executed again on the first data through the first thread.
Optionally, the device in the embodiment of the present application may further include: a persistence module, configured to perform persistence processing on the data in the retry queue under a condition that at least one of the following is satisfied: before the processing process of the message corresponding to the data in the retry queue is stopped, the data volume of the data in the retry queue is larger than or equal to a preset threshold value; wherein, the persistence processing refers to the data being transferred to the memory medium.
Optionally, the device in the embodiment of the present application may further include: and the reporting module is used for reporting a first notification message to a third server after the data in the retransmission queue is subjected to persistence processing, wherein the first notification message is used for indicating that the write operation is abnormal, and the third server is a server in the alarm platform.
Optionally, the device in the embodiment of the present application may further include: and the deleting module is used for deleting the first data from the retry queue under the condition that the first processing operation is successfully executed on the first data again through the first thread.
Optionally, the device in the embodiment of the present application may further include: the execution module is used for re-executing the first processing operation before the first server writes the first data corresponding to the first message into the retry queue; the second processing module is used for writing the first data into the log and saving the log to the disk under the condition that the re-execution of the first processing operation fails; and the third processing module is used for counting the abnormal attributes of the logs in the disk and reporting a second notification message to the third server according to the counting result, wherein the second notification message is used for notifying that the write operation is abnormal.
The embodiment of the application also provides an electronic device, as shown in fig. 12, which comprises a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202 and the memory 1203 complete the communication with each other through the communication bus 1204,
a memory 1203 for storing a computer program;
the processor 1201, when executing the program stored in the memory 1203, performs the following steps:
step 11, under the condition that the execution of the first processing operation corresponding to the first message fails, the first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to a first server consuming the message by a second server producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
in step 12, the first server starts a first thread and re-executes a first processing operation via the first data in the first line Cheng Duichong queue, wherein the first thread is different from the second thread based on the second thread performing a plurality of processing operations.
The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the terminal and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present application, a computer readable storage medium is provided, in which instructions are stored, which when run on a computer, cause the computer to perform the method for processing data according to any of the above embodiments.
In a further embodiment of the present application, a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of processing data according to any of the above embodiments is also provided.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims (9)
1. A method of processing data, comprising:
under the condition that the execution of a first processing operation corresponding to a first message fails, the first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to the first server consuming the message by a second server producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
before re-executing the first processing operation on the first data by the first thread, and in the case that the first data also corresponds to the second message, the first server determines an execution state of the second processing operation corresponding to the second message; wherein the execution result of the second processing operation will override the execution result of the first processing operation; the execution time of the second processing operation is later than the execution time of the first processing operation;
under the condition that the second processing operation fails to be executed, the first server writes first data corresponding to the second message into the retry queue;
the first server prohibiting the first thread from being started if the second processing operation is successful and the first processing operation has not been re-performed on the first data by the first thread;
the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue by the first thread, wherein the plurality of processing operations are executed based on a second thread, the first thread being different from the second thread.
2. The method of claim 1, wherein the first server writing the first data corresponding to the first message to a retry queue comprises:
the first server writes the first data and a timestamp corresponding to the first data into the retry queue, wherein the timestamp is execution time of the first processing operation.
3. The method according to claim 1, wherein the method further comprises:
the first server performs persistence processing on the data in the retry queue on a condition that at least one of: before the processing process of the message corresponding to the data in the retry queue is stopped, the data volume of the data in the retry queue is larger than or equal to a preset threshold value;
wherein, the persistence processing refers to the data being transferred to a memory medium.
4. The method of claim 3, wherein after the first server persists the data in the retry queue, the method further comprises:
the first server reports a first notification message to a third server, wherein the first notification message is used for indicating that the writing operation is abnormal, and the third server is a server in the alarm platform.
5. The method according to claim 1, wherein the method further comprises:
the first server deletes the first data from the retry queue if the re-execution of the first processing operation on the first data by the first thread is successful.
6. The method of claim 1, wherein before the first server writes the first data corresponding to the first message to a retry queue, the method further comprises:
the first server re-performs a first processing operation;
the first server writes the first data into a log and stores the log to a disk under the condition that the re-execution of the first processing operation fails;
the first server counts the abnormal attributes of the logs in the disk, and reports a second notification message to a third server according to the statistical result, wherein the second notification message is used for notifying that the write operation is abnormal.
7. A data processing apparatus, applied to a first server, comprising:
the first writing module is used for writing first data corresponding to a first message into a retry queue under the condition that the execution of a first processing operation corresponding to the first message fails, wherein the first message is sent to the first server for consuming the message by a second server for producing the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
a determining module, configured to determine, before the first processing operation is re-performed on the first data by the first thread, and in a case where the first data also corresponds to the second message, an execution state of the second processing operation corresponding to the second message; wherein the execution result of the second processing operation will overlap the execution result of the first processing operation; the execution time of the second processing operation is later than the execution time of the first processing operation;
the second writing module is used for writing the first data corresponding to the second message into the retry queue under the condition that the execution of the second processing operation fails;
the prohibiting module is used for prohibiting starting the first thread under the condition that the second processing operation is successful and the first processing operation is not executed again on the first data through the first thread;
and the first processing module is used for starting a first thread and re-executing the first processing operation on the first data in the retry queue through the first thread, wherein the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
8. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1-6 when executing a program stored on a memory.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475234.1A CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475234.1A CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113176967A CN113176967A (en) | 2021-07-27 |
| CN113176967B true CN113176967B (en) | 2023-08-15 |
Family
ID=76925315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110475234.1A Active CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113176967B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| CN107688503A (en) * | 2017-09-07 | 2018-02-13 | 北京奇艺世纪科技有限公司 | A kind of message treatment method based on ActiveMQ data/address bus, device and electronic equipment |
| CN109308214A (en) * | 2017-07-27 | 2019-02-05 | 北京京东尚科信息技术有限公司 | Data task processing method and system |
| CN112099935A (en) * | 2020-09-25 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Task processing method and device |
| CN112148505A (en) * | 2020-09-18 | 2020-12-29 | 京东数字科技控股股份有限公司 | Data batching system, method, electronic device and storage medium |
| CN112328418A (en) * | 2020-11-27 | 2021-02-05 | 行吟信息科技（上海）有限公司 | Method and system for improving MQ synchronization reliability |
| CN112395097A (en) * | 2019-08-14 | 2021-02-23 | 北京京东尚科信息技术有限公司 | Message processing method, device, equipment and storage medium |
| CN112685145A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8549538B2 (en) * | 2010-03-18 | 2013-10-01 | Microsoft Corporation | Coordinating communication medium state for subtasks |
| US9575820B2 (en) * | 2014-06-27 | 2017-02-21 | Amazon Technologies, Inc. | Client control in a distributed strict queue |
| US10819648B2 (en) * | 2017-06-27 | 2020-10-27 | Atlassian Pty Ltd. | Retry handling in messaging queues |
-
2021
- 2021-04-29 CN CN202110475234.1A patent/CN113176967B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| CN109308214A (en) * | 2017-07-27 | 2019-02-05 | 北京京东尚科信息技术有限公司 | Data task processing method and system |
| CN107688503A (en) * | 2017-09-07 | 2018-02-13 | 北京奇艺世纪科技有限公司 | A kind of message treatment method based on ActiveMQ data/address bus, device and electronic equipment |
| CN112395097A (en) * | 2019-08-14 | 2021-02-23 | 北京京东尚科信息技术有限公司 | Message processing method, device, equipment and storage medium |
| CN112685145A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
| CN112148505A (en) * | 2020-09-18 | 2020-12-29 | 京东数字科技控股股份有限公司 | Data batching system, method, electronic device and storage medium |
| CN112099935A (en) * | 2020-09-25 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Task processing method and device |
| CN112328418A (en) * | 2020-11-27 | 2021-02-05 | 行吟信息科技（上海）有限公司 | Method and system for improving MQ synchronization reliability |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113176967A (en) | 2021-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107908494B (en) | Abnormal event processing method and device, electronic equipment and storage medium | |
| US9419650B2 (en) | Flexible event data content management for relevant event and alert analysis within a distributed processing system | |
| US8639980B2 (en) | Administering incident pools for event and alert analysis | |
| US8880943B2 (en) | Restarting event and alert analysis after a shutdown in a distributed processing system | |
| US7747717B2 (en) | Fast application notification in a clustered computing system | |
| US8689050B2 (en) | Restarting event and alert analysis after a shutdown in a distributed processing system | |
| CN110941502B (en) | Message processing method, device, storage medium and equipment | |
| CN111585867A (en) | Message processing method and device, electronic equipment and readable storage medium | |
| CN111949497B (en) | Message queue system and message processing method based on same | |
| CN112527472A (en) | Distributed transaction processing system and method based on microservice | |
| CN111010318A (en) | Method and system for discovering loss of connection of terminal equipment of Internet of things and equipment shadow server | |
| CN111782440A (en) | Data processing method and device and electronic equipment | |
| CN111258790A (en) | Anomaly compensation method and device | |
| CN113176967B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US11243712B2 (en) | Local analytics for high-availability storage systems | |
| CN115037653B (en) | Service flow monitoring method, device, electronic equipment and storage medium | |
| CN111400085A (en) | Transaction rollback exception processing method and device, computer equipment and storage medium | |
| US11704180B2 (en) | Method, electronic device, and computer product for storage management | |
| CN112711466B (en) | Hanging affair inspection method and device, electronic equipment and storage medium | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN111639089B (en) | Transaction processing method, transaction processing device, electronic equipment and computer readable storage medium | |
| CN113703669B (en) | Cache partition management method, system, equipment and storage medium | |
| JP6674916B2 (en) | Communication fault management device and communication system | |
| CN115344437A (en) | Disaster tolerance switching method and device, electronic equipment and storage medium | |
| CN117407282A (en) | Application program warning method, device, equipment, storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |