WO2013143050A1

WO2013143050A1 - Service processing method, execution unit and system for distributed operating system

Info

Publication number: WO2013143050A1
Application number: PCT/CN2012/073003
Authority: WO
Inventors: 曹俊
Original assignee: 华为技术有限公司
Priority date: 2012-03-26
Filing date: 2012-03-26
Publication date: 2013-10-03
Also published as: CN104145260A; CN104145260B

Abstract

A service processing method, execution unit and system for a distributed operating system. The method comprises the following operations executed by the execution unit in each service step: as for an acquired message to be processed, a current execution unit launches a service processing program of the service step to generate a service processing result message; the generated service processing result message is stored in a corresponding temporary queue; it is determined whether there are no other execution units that store the result message of service processing produced by processing the message to be processed in the temporary queue corresponding to other execution units. If no, the temporary queue corresponding to the current execution unit is determined as the current temporary queue, and all the service processing result messages of said current temporary queue are transferred to an output queue; If yes, it is determined whether said message to be processed has been fully processed by other execution units.

Description

Service processing method, execution unit and system of distributed operation system

The present invention relates to the field of distributed technologies, and in particular, to a service processing method, an execution unit, and a system of a distributed operating system.

Background technique

The distributed operating system is a solution for solving large-scale computing by executing jobs in parallel by multiple hosts, and is widely used in data mining, media processing, and scientific research.

Currently, Amazon Simple Queue Service (SQS) provides a scalable and reliable messaging framework that is a great way to build a loosely coupled distributed operating system. The distributed operation system based on SQS technology transfers and distributes task information to each execution host of the system through SQS queue, and realizes large-scale calculation by executing jobs in parallel.

In a distributed operating system, sometimes a failure recovery of a host is performed due to a failure or a mechanism in which the SQS itself allows multiple processing hosts to process the same message, resulting in frequent output of duplicate messages. The prior art also attempts to alleviate this problem to some extent.

For example, in the Cloud MapReduce technology, in the processing of the Map subtask, multiple processing hosts are allowed to process the same input message, and the message is output to the same Reduce queue. The Reduce queue is used as the input of the Reduce subtask, and the output message contains three identifiers. : Executes the host ID, mapID, and message ID, and writes the processing record of the Ma subtask to SDB (SimpleDB) before output. In the Reduce subtask, after the processing host obtains the message from the Reduce queue, the three labels of the message are recorded in the SDB, and the SDB is queried for the execution host ID list corresponding to the MapID. If there are multiple execution host IDs, The output of one of the execution host IDs is selected, and the output of the other execution host IDs is discarded, so that the duplicate messages output by the Map subtask are removed, and are no longer brought into the processing of the Reduce subtask below.

However, the solution for extracting duplicate messages proposed in the above prior art is specific to a specific task, for example, for the output of the Map subtask, but not applicable to other subtasks, or the setting does not propose any solution, such as in Cloud MapReduce technology. There is no proposal to solve the problem of the Reduce subtask outputting duplicate messages.

In summary, the distributed operating system in the prior art does not exist in the process of implementing business processing. A general method of removing duplicate messages.

Summary of the invention

In view of this, an object of the embodiments of the present invention is to provide a service processing method, an execution unit, and a system for a distributed operating system, which are applicable to the purpose of removing duplicate messages in any task in the multitasking process, and have good versatility.

To achieve the above objective, an embodiment of the present invention provides a service processing method for a distributed operating system, where an execution unit in each service step performs the following operations for each pending message retrieved from an input queue of the service step. :

The current execution unit runs a service processing program of the service step where the execution unit is located for a message to be processed obtained from the input queue, and generates at least one service processing result message;

The current execution unit stores the generated service processing result message in a temporary queue corresponding to the current execution unit;

Determining whether there is no other execution unit to store the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, and if not, the temporary queue corresponding to the current execution unit is used as the current a temporary queue, and the current execution unit transfers all the service processing result messages in the current temporary queue to the output queue, so that the to-be-processed message is completely processed by the current execution unit, and if so, the pending message is determined. Whether it has been completely processed by other execution units. For the case where the processing is not completely processed, the temporary queue of the other execution unit that has been stored in the temporary processing queue for processing the processed message is processed as the current Temporary queue.

In another aspect, the present invention further provides an execution unit of a distributed operating system, configured to process, in a service step in which it is located, each pending message retrieved from an input queue, the execution unit comprising:

a service processing unit, configured to: for a pending message obtained from the input queue, run a service processing program of the service step where the execution unit is located, generate at least one service processing result message; and process a result saving unit, which is to be generated The service processing result message is stored in a temporary queue corresponding to the execution unit; The first processing result is stored in the judging unit, and is configured to determine whether the other execution unit has stored the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, if not, The temporary queue corresponding to the current execution unit is used as the current queue, and if so, the processing completion judgment unit is triggered to work;

The processing completion judging unit is configured to determine whether the to-be-processed message is completely processed by other execution units, and if the incomplete processing is completed, the pending queue of the other execution unit is already stored in the pending message. The temporary queue for processing the obtained service processing result message is used as the current temporary queue;

The processing result transfer unit is configured to transfer all the service processing result messages in the current temporary queue to the output queue, and the pending message is completely processed.

Another aspect of the present invention also provides a distributed operating system comprising a plurality of the above-described execution units and a storage unit storing a message record table.

It can be seen that, in the embodiment of the present invention, before each execution unit transfers the service processing result message processed for one to-be-processed message to the output queue, it is determined whether there is no other execution unit that will process the pending item. The service processing result message obtained by the message is stored in the corresponding temporary queue. If the judgment result is not, the current execution unit will transfer the service processing result message outputted by the current execution unit to the output queue. Otherwise, the current execution unit The business process result message that was previously processed by other execution units is transferred to the output queue. This ensures that for the same pending message in the input queue, only one execution unit processes the output of the service processing result message and finally sends it to the output queue, avoiding outputting duplicate service processing results to the output queue for the same pending message. Message. And the solution can be applied to the various business steps of the distributed business system, so it is more versatile.

DRAWINGS

1 is a flowchart of a service processing method of a distributed operating system according to Embodiment 1 of the present invention; FIG. 2 is a flowchart of a service processing method of a distributed operating system according to Embodiment 2 of the present invention; A schematic diagram of a scenario application scenario in Example 3;

4 is a schematic diagram of a processing procedure of an execution unit of an Nth service step in Embodiment 3 of the present invention; FIG. 5 is a flowchart of a service processing method of a distributed operating system according to Embodiment 3 of the present invention; 6 is a schematic diagram of an application example of a service processing method of a distributed operating system according to an embodiment of the present invention;

Figure Ί is a schematic diagram of another application example of the service processing method of the distributed operating system provided by the embodiment of the present invention;

FIG. 8 is a schematic diagram of another application example of a service processing method of a distributed operation system according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic diagram of another application example of a service processing method of a distributed operating system according to an embodiment of the present invention; FIG.

FIG. 10 is a schematic diagram of an execution unit of a distributed operating system according to an embodiment of the present invention; FIG. 11 is a schematic diagram of an execution unit of a distributed operating system according to another embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A first embodiment of the present invention provides a service processing method for a distributed operating system. As shown in FIG. 1, in the method, an execution unit in each service step is for each item to be taken from an input queue of the service step. Processing the message performs the following operations:

Step S101: The current execution unit runs at least one service processing result message for the service processing program of the service step where the execution unit is located, for a message to be processed obtained from the input queue.

Step S102: The current execution unit stores the generated service processing result message in a temporary queue corresponding to the current execution unit.

Step S103: determining whether there are other execution units storing the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, and if not, using the temporary queue corresponding to the current execution unit as The current temporary queue is executed, and step S105 is performed; if yes, step 104 is performed.

Step S104: It is determined whether the to-be-processed message has been completely processed by other execution units. For the case that the processing is not completely processed, the service processing result of processing the pending message in the temporary queue of the other execution unit is already stored. The temporary queue of the message acts as the current temporary queue, then Step S105 is performed.

Step S105: The current execution unit transfers all the service processing result messages in the current temporary queue to the output queue, and the to-be-processed message is completely processed by the current execution unit.

It can be seen that, in the method provided by the embodiment of the present invention, before each execution unit transfers the service processing result message processed for one to-be-processed message to the output queue, it is determined whether no other execution unit will process the The service processing result message obtained by the message to be processed is stored in the corresponding temporary queue. If the judgment result is not, the current execution unit will transfer the service processing result message outputted by the current execution unit to the output queue. Otherwise, The current execution unit transfers the business process result message that was previously processed by other execution units to the output queue. This ensures that for the same pending message in the input queue, only one execution unit processes the output of the service processing result message and finally sends it to the output queue, avoiding outputting duplicate service processing results to the output queue for the same pending message. Message.

It should be noted that the method provided by the embodiment of the present invention is a solution that can be applied to various service steps of a distributed service system, and therefore is more versatile.

Embodiment 2

In the solution of the first embodiment, although it is guaranteed that the duplicate service processing result message is not output to the output queue for the same pending message, it may happen that: in one service step, one execution unit A. Before executing the service processing program in step S101 for a message to be processed, there may already be other execution units (for example, execution unit B) executing the same business processing program for the message to be processed, and the processed business has been processed. The result message is stored in the corresponding temporary queue, but since the execution unit A determines that the execution unit B has processed the message when executing to step S103, and has already stored the output message in the temporary queue of the execution unit B, Therefore, the execution unit A still performs step S101 and step S102 to execute the business processing program on the same pending message, which actually causes waste of processing resources.

In order to avoid the waste of the foregoing processing resources that may be caused by the method in the first embodiment, an optimized solution is proposed in the second embodiment of the present invention. As shown in FIG. 2, in the method, the execution unit in each service step is from the The following steps are taken to retrieve each pending message from the input queue of the business step:

Step S201: determining whether there is no other execution unit to store the service processing result outputted by the current execution unit to be processed into the temporary queue corresponding to the other execution unit, such as If not, step 202 is performed; if yes, step S205 is performed.

Step S202: The current execution unit runs a service processing procedure of the service step of the execution unit for the acquired to-be-processed message, and generates at least one service processing result message.

Step S203: The current execution unit stores the generated service processing result message in a temporary queue corresponding to the current execution unit.

Step S204: determining whether there are other execution units storing the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, and if not, using the temporary queue corresponding to the current execution unit as The current temporary queue is executed, and step S206 is performed; if yes, step 205 is performed.

Step S205: It is determined whether the to-be-processed message has been completely processed by other execution units. For the case that the processing is not completely processed, the service processing result of processing the pending message in the temporary queue of the other execution unit is already stored. The temporary queue of the message is used as the current temporary queue, and then step S206 is performed.

Step S206: The current execution unit transfers all the service processing result messages in the current temporary queue to the output queue, and the to-be-processed message is completely processed by the current execution unit.

Therefore, the method provided in this embodiment can not only output a repeated service processing result message to the output queue for the same pending message, but also determine whether the current execution unit performs a service processing procedure on a to-be-processed message. There is no other execution unit that stores the service processing result message outputted by the processing of the to-be-processed message in the temporary queue corresponding to the other execution unit. If the judgment result is not, the current execution unit will process the item to be processed. Messages are processed to avoid wasting processing resources. Embodiment 3

In this example, the implementation of the solution of the present invention is described in detail in a specific application scenario.

In the distributed operating system to which the solution in this embodiment is applied, as shown in FIG. 3, the entire service model is composed of multiple service steps (steps), and the relationships between the business steps are concatenated, as shown in FIG. Each business step consists of an input queue, an execution unit, an output queue, and a message log table to record the processing of the message.

The StepN queue in Figure 3 refers to the input queue of the execution unit of the Nth business step. For the message to be processed, it is also the output queue of the execution unit of the N-1th business step for collecting and transmitting the generated new message.

The execution unit of each business step is the smallest execution unit in the business, and each business step can have multiple execution units. The execution unit gets the message from the input queue, runs the business program to process the message, and puts the generated message into the output queue.

The message record table can be stored in the SDB, which can be a non-relational data storage database that can be scaled and highly available. In the embodiment of the present invention, the structure of the data is specially designed. For example, each row of data is an item, each entry may have multiple attributes, and each attribute may have multiple values (value ).

In the message record table of the embodiment of the present invention, the message ID (MsgID) of each to-be-processed message is taken as an entry, and the attributes in each entry include the receiving time (ReceiveTime), the execution unit ID (WorkerDD), and the commit time (CommitTime). ), temporary queue address ( TempQueueURL ) and completion time ( FinishTime ).

When processing a message in each execution unit, a temporary queue is created. All new messages generated during message processing are placed in the temporary queue, and the address of the temporary queue is written in the message record table. The new message corresponds to the entry in the "temporary queue address" attribute.

The writing conditions of the message recording table are specifically limited in the present invention, and will be described in detail in the subsequent steps.

FIG. 4 is a schematic diagram showing the processing procedure of the execution unit of the Nth service step, and it can be seen that the execution unit of the Nth service step outputs the queue from the execution unit of the N-1th business step (also the Nth business step). Obtaining a message in the input queue of the execution unit (step 1 in Fig. 4), and writing a message acquisition record for the acquired message in the message record table (step 2 in Fig. 4); when the Nth service After the processing unit of the step finishes processing the message in the third step, the generated new message is placed in the temporary queue of the execution unit, and the message processing record is written in the fourth step; then the Nth business step is The execution unit transfers the message in the temporary queue to the output queue (step 5 in Figure 4) and writes the completion record in the message record table (step 6 in Figure 4). It should be noted that, in FIG. 4, the execution of an execution unit of the Nth service step in the case that the writing step is normally completed, in fact, in the steps of the method provided by the embodiment of the present invention, Consider the conditional constraints of writing to achieve the problem of preventing duplicate message output, as detailed below. Referring to FIG. 5, the service processing method of the distributed operating system provided by this embodiment includes the following steps:

Step S501: The execution unit acquires a message from the input queue as the currently processed message. In this embodiment, the execution unit is not limited to an execution unit at a specific service step. When the execution unit is the execution unit of the second service step and the subsequent step, the input queue is a queue formed by the message output by the previous service step of the service step where the execution unit is located; when the execution unit is the first service When the execution unit of the step, the input queue is the queue formed by the message originally received by the system.

Step S502: The execution unit starts to write a message acquisition record to the message record table by using the acquired ID of the to-be-processed message, and the process of writing includes: to the item corresponding to the ID of the acquired message (Item) The receiving time " attribute joins the execution unit to get the message, and adds the ID of the execution unit to the "execution unit ID" attribute in the same entry.

Step S503: determining whether the value of the "commit time" attribute in the entry written in step S502 is empty, if yes, the execution unit succeeds in step S502, and proceeds to step S504; otherwise, indicates the execution unit The writing in step S502 fails, the data written in step S502 is cleared, and then step 510 is performed.

In the present embodiment, it is judged whether or not the value of the "commit time" attribute in the written entry is empty. Actually, it is judged whether or not the message processing record has been written in the entry.

Step S504: After confirming that the writing is successful in step S503, the executing unit runs the business program to process the acquired pending message, and stores the new message generated during the processing (ie, the business processing result message) into the execution unit. In the temporary queue.

In fact, since the "commit time" in an entry is that the message corresponding to the entry has been processed by an execution unit and the business process result message is stored in the temporary queue, it is added, so when the step When it is confirmed in S503 that the writing is successful, it indicates that the value of the "commit time" attribute in the entry of the acquired message is null, further indicating that the pending message has not been successfully processed by other execution units and the business processing result message is stored. It is meaningful to the temporary queue, so the execution unit continues to process the obtained message, which can further save processing resources.

Step S505: Writing a message processing record with the ID of the message to be processed, the process of writing includes: starting a "commit time" and a "temporary queue address" in an entry corresponding to the ID of the to-be-processed message. The two attributes are respectively added to the time when the generated service processing message is stored in the temporary queue in step S504 and the address of the temporary queue.

Step S506: It is judged whether the value of the "commit time" attribute in the entry written in step S505 is empty. If yes, the execution unit succeeds in the writing in step S505, and the temporary queue address written in step S505 is obtained. As the current temporary queue address, step S507 is continued; otherwise, the write fails, and the data written in step S505 is cleared, and then step S510 is performed.

In fact, for the reason that it is determined in step S503 that the acquisition message record is successfully written, but the subsequent step S506 is still based on the similar submission to determine whether the processing message is successfully written, the reason is: at a certain execution unit in step S502 When the message record is successfully written in the acquisition, no other execution unit has processed the acquired message, that is, the process is judged to be successful in step S503, but in the process of processing the acquired message by the execution unit, There may be other execution units that process the acquired same message and store it in the temporary queue, thereby updating the entry in the message record table about the message, so that the attributes of "commit time" and "temporary queue address" are both It is no longer empty. If it is no longer judged according to the same write restriction condition in the subsequent step S506, it will cause repeated processing and finally output a new message that is repeated.

Step S507: Obtain a message from the corresponding temporary queue according to the current temporary queue address (that is, obtain each service processing result message), and put it into the output queue until the temporary queue is emptied.

Step S508: Write the write completion record in the entry to the entry corresponding to the message ID of the to-be-processed message in the message record table, and specifically include: transferring all the service processing messages in the current temporary queue to the output queue The time is written to the "Completion Time" attribute in the entry.

Step S509: The currently processed message is deleted from the input queue, and the process returns to step S501.

Step S510: When the result of the determination in step S503 or step S506 is no, the value of all the attributes in the entry of the current message to be processed is obtained from the message record table.

In fact, when an execution unit executes step S510, it means that the other execution units have processed the same message in the input queue, but whether the generated new message has been completely put into the output queue is determined by the subsequent judgment step.

Step S511: It is judged whether the value of the "Completion Time" attribute in the attribute obtained in step S510 is empty, and if not, the process returns to step S501; if yes, the process proceeds to step S512.

Step S512: The value of the "temporary queue address" attribute in the entry of the current pending message As the current temporary queue address, then return to step S507.

In fact, if the result of the determination in step S511 is NO, it indicates that the same pending message has been completely processed by other execution units, and the generated service processing result message has been output to the output queue, so the execution unit should not re-book this The message is subsequently processed, and the next message should be retrieved from the input queue again, that is, the process returns to step S501.

If the result of the determination in step S511 is YES, it indicates that the other execution units have finished processing the same message in the input queue, and the new message outputted after the processing has been written into the temporary queue of the other execution unit, and is also written. The message processing record. At this time, according to step S512, the current execution unit calls up the temporary queue address of the execution unit that previously wrote the message processing record, and the address of the temporary queue is used as the address of the current temporary queue, and then proceeds to the temporary queue by performing step S507. The new message that has been output is placed in the output queue of this business step, thereby preventing multiple execution units from repeatedly outputting the same new message. The above is the basis of describing the steps of the present invention in terms of a certain execution unit in each of the business steps in the embodiment of the present invention with reference to Fig. 5, and the specific application of the plurality of execution unit cases will be described in detail below.

Figure 6 shows the entire process of obtaining a message from an input queue with only one execution unit in the Nth business step. As shown in FIG. 6, the process specifically includes:

Step S601: The execution unit acquires a pending message from the input queue.

Step S602: The execution unit writes a message acquisition record to the to-be-processed message in the message record table. Since the message is obtained by the execution unit from the input queue for the first time, there may not be any message in the message record table. An entry, so in this step, a new entry is created for the acquired message, and the values of the two attributes of the receiving time and the execution unit ID contained in the message acquisition record are filled in.

Step S603: Since no other execution unit acquires the message before, in the message record table, the value of the submission time attribute of the entry corresponding to the message is null, so the message record table is successfully written, and the execution unit is notified.

Step S604: The execution unit acquires a business processing program of the business step of the message to be processed for the strip, and stores the output new message in the temporary queue.

Step S605: The execution unit starts to input an entry for the to-be-processed message in the message record table. Writes a message processing record, including the write commit time and the temporary queue address where the output message is stored.

Step S606: It is determined in step S605 that the write message processing record is successfully written, and the execution unit is notified.

Step S607: The execution unit moves all the messages in the temporary queue to the output queue.

Step S608: In the execution unit item message record table, the message completion record is written for the entry corresponding to the to-be-processed message, that is, the value of the completion time attribute is added to the entry.

Step S609: It is determined that the write message completion record is successfully written, and the execution unit is notified.

Step S610: The execution unit deletes the previously processed message from the input queue. Figure 7 shows the overall processing of multiple execution units getting the same message from the input queue in the Nth business step. Due to the nature of the SQS mechanism, multiple execution units that acquire messages from the same input queue may acquire the same message, and in the SQS mechanism, allow multiple execution units to simultaneously process the same message, new messages generated in the present invention. They are all put into their respective temporary queues, but through the conditional write mechanism illustrated in Figure 5, only one execution unit's output message processed by the message will be placed in the output queue, ensuring that there will be no duplicate processing results. Put in the output queue.

As shown in Figure 7, the process specifically includes:

Step S701: The execution unit 1 obtains the message a from the input queue.

Step S702: The execution unit 1 writes a message acquisition record to the message record table, and further creates an entry for the message a in the message record table, and two attributes of "reception time" and "execution unit ID" in the entry. The time is added to the execution unit 1 to obtain the message a and the ID of the execution unit 1, respectively, and the message is obtained by the message acquisition record writing condition (that is, the value of the "commit time" attribute in the entry of the message a is empty), and the writing is determined. success.

Step S703: After that, the execution unit 2 acquires the same message a from the input queue.

Step S704: The execution unit 2 also starts writing a message acquisition record to the message record table, that is, adding the time of the execution unit 2 to the message a to the two attributes of the "reception time" and the "execution unit ID" of the entry of the message a And the ID of the execution unit 2; since the value of the "commit time" attribute in the entry of the message a is still empty in the message record table at this time, the write operation of the write message acquisition record of the execution unit 2 is also satisfied. Write conditions to determine that the write was successful. Therefore, at this time, the value of the execution unit ID attribute in the entry of the message a includes two: the ID of the execution unit 1 and the ID of the execution unit 2; and the value of the "reception time" attribute of the entry also has two: execution unit 1 The time at which the message a is acquired and the time at which the execution unit 2 acquires the message a.

Step S705: The service program running the Nth business step of the execution unit 1 processes the message a obtained, and generates a new message to be stored in the temporary queue 1.

Step S706: The execution unit 2 also processes the message A of the Nth business step to process the message a obtained, and generates a new message to be stored in the temporary queue 2.

It should be noted that, since the method provided by the embodiment of the present invention is a service processing method of a distributed operation system, the service programs run by multiple execution units in the same service step are the same, so the execution unit 1 and the execution are performed. The new message output by unit 2 processing the same message a is also the same (number of messages, content and type).

Step S707: Since the execution unit 1 completes the processing of the message a first, the execution unit 1 first starts writing the message processing record to the message record table, that is, the "submission time" in the entry corresponding to the message a in the message record table. "The time at which the join execution unit 1 stores the new message obtained by processing the message a into the temporary queue 1, and the address of the temporary queue 1 in which the new message of the execution unit 1 is stored in the "temporary queue address" attribute is added; When the unit 1 writes the message processing record for the message a, the value of the "commit time" attribute in the entry of the message a is still empty, so the writing condition is satisfied, and the writing is confirmed to be successful (step S708).

Step S709: After confirming that the message processing record is successfully written, the execution unit 1 acquires the message from the temporary queue 1 and starts to transfer to the output queue.

Step S710: When the execution unit 1 has not completely transferred all the messages in the temporary queue 1 to the output queue, the execution unit 2 starts writing the message processing record to the message record table, that is, the entry corresponding to the message a in the SDB. The "commit time" in the joining execution unit 2 stores the new message obtained by processing the message a into the temporary queue 2, and adds the temporary queue 2 of the new message that the execution unit 2 stores the output to the "temporary queue address" attribute. Address; Since the message processing record is written to the message a at the execution unit 2, the value of the "commit time" attribute in the entry of the message a has been added to the execution unit 1 in step S707, and the resulting new message is stored in the temporary queue 1 Since the write condition is not satisfied in this step, the write failure is confirmed (step S711).

Step S712: The execution unit 2 receives the failure of the message processing record write failure for the message a. After knowing, the value of each attribute in the entry of the message a is read from the message record table, the value of the "temporary queue address" attribute is obtained, and the address of the temporary queue 1 represented by the value is taken as the temporary to be operated. Queue address.

Step S713: The execution unit 2 continues to obtain the remaining messages from the temporary queue 1 according to the address of the temporary queue 1, and transfers to the output queue.

It should be noted that, in this embodiment, multiple execution units are allowed to transfer messages in the same temporary queue.

Step S714: The execution unit 1 writes a message completion record to the entry of the message record table for the message a, and the writing is successful.

Step S715: The execution unit 2 writes a message completion record to the entry of the message record table for the message a, and the writing is successful.

In this embodiment, it is only necessary to ensure that the message transferred to the output queue is only in one temporary queue, and the message completion record does not need to be unique.

Step S716: The execution unit 1 deletes the message a from the input queue.

Step S717: Execution unit 2 deletes the message from the input queue.

In this embodiment, if the message a has been successfully deleted in step S716, the deletion action in step S717 is still performed, but only when the actual deletion is found, the message a does not exist, but the result of the successful deletion can still be returned.

It can be seen that although due to the mechanism of SQS itself, multiple execution units will acquire the same message and output the same new message, and then store them in different temporary queues, but only one message in the temporary queue will be transferred to the output. In the queue, there is no problem with duplicate processing results for the same message output. In addition, the execution unit will inevitably fail, and the failure may cause the acquired message to be unsuccessfully processed, or may cause the output to repeat the message due to the inappropriate failure recovery method. In the following, the specific process of using the conditional write mechanism illustrated in FIG. 5 will be described in detail by taking the execution unit as a failure during the message processing and after the end of the message processing.

FIG. 8 is a flowchart of implementing message reprocessing after a certain execution unit in the Nth service step fails in the message processing process in the embodiment of the present invention, which specifically includes: Step S801: The execution unit 1 acquires the message a from the input queue.

Step S802: The execution unit 1 writes a message acquisition record to the message record table, and further creates an entry for the message a in the message record table, and two attributes of "reception time" and "execution unit ID" in the entry. The time is added to the execution unit 1 to obtain the message a and the ID of the execution unit 1, respectively, and the message is obtained by the message acquisition record writing condition (that is, the value of the "commit time" attribute in the entry of the message a is empty), and the writing is determined. success.

Step S803: The execution unit 1 starts the business program running the Nth business step to process the message a obtained, and simultaneously stores the generated new message into the temporary queue 1.

Step S804: The execution unit 1 fails during the execution of the business program.

Step S805: Execution unit 2 acquires the same message a from the input queue.

Step S806: The execution unit 2 also starts to write a message acquisition record to the message record table, that is, adding the execution unit 2 to the two items of the "reception time" and the "execution unit ID" of the entry of the message a to obtain the message a. Time and ID of execution unit 2; since the value of the "commit time" attribute in the entry of the message a is still empty in the message record table at this time, the write operation of the write message acquisition record of the execution unit 2 is also The write condition is met and the write is determined to be successful.

Step S807: The execution unit 2 also runs the business program of the Nth business step to process the message a obtained, and generates a new message to be stored in the temporary queue 2.

Step S808: After processing the message a and storing all the generated new messages into the temporary queue 2, the execution unit 2 starts writing the message processing record to the message record table, that is, corresponding to the message a in the message record table. The "commit time" in the entry joins the execution unit 2 to store the new message obtained by processing the message a into the temporary queue 2, and the temporary queue for adding the new message of the execution unit 2 to the output of the "temporary queue address" attribute. Address of 2; since the value of the "commit time" attribute in the entry of the message a is still empty in the message record table at this time, the write operation of the write message processing record of the execution unit 2 also satisfies the write condition , to determine the success of the write.

Step S809: After the acknowledgment message processing record is successfully written, the execution unit 2 transfers all the messages in the temporary queue 2 of the execution unit 2 to the output queue.

Step S810: After the execution unit 2 transfers all the messages in the temporary queue 2, write a message completion record to the message record table for the message a, that is, write to the "Completion Time" attribute in the entry corresponding to the message a. Execution unit 2 transfers the time in temporary queue 2. Step S811: The execution unit 2 deletes the processed message a from the input queue.

It can be seen that when an execution unit obtains a message from the input queue, if a failure occurs during processing, the unprocessed message will reappear in the input queue due to the message processing timeout replay mechanism of the SQS. . In turn, another execution unit will get this message and re-process it to ensure that the business can continue. At the same time, due to the condition limitation of the write record in the embodiment of the present invention, only the processing result in the temporary queue is transferred to the output queue for the processing of the same message, thereby avoiding the repeated processing result for the processing output of the same message. The problem.

FIG. 9 is a flow chart showing the re-processing of a message after a failure of an execution unit in the Nth service step after the end of the message processing in the embodiment of the present invention, specifically including:

Step S901: The execution unit 1 acquires the message a from the input queue.

Step S902: The execution unit 1 writes a message acquisition record to the message record table, and further creates an entry for the message a in the message record table, and two attributes of "reception time" and "execution unit ID" in the entry. The time is added to the execution unit 1 to obtain the message a and the ID of the execution unit 1, respectively, and the message is obtained by the message acquisition record writing condition (that is, the value of the "commit time" attribute in the entry of the message a is empty), and the writing is determined. success.

Step S903: The execution unit 1 starts the business program running the Nth business step to process the message a obtained, and simultaneously stores the generated new message into the temporary queue 1.

Step S904: After processing the message a and storing all the generated new messages into the temporary queue 1, the execution unit 1 starts writing the message processing record to the message record table, that is, corresponding to the message a in the message record table. The "commit time" in the entry joins the time when the execution unit 1 stores the new message obtained by processing the message a into the temporary queue 1, and the temporary queue that adds the new message of the execution unit 1 to the output to the "temporary queue address" attribute. Address of 1; Since the value of the "commit time" attribute in the entry of the message a is still empty when the execution unit 1 writes the message processing record for the message a, the write condition is satisfied, and the write is confirmed to be successful.

Step S905: After confirming that the message processing record is successfully written, the execution unit 1 acquires a message from the temporary queue 1 of the execution unit 1, and starts to transfer to the output queue.

Step S906: In the process of transferring the message, the execution unit 1 fails, and the message in the temporary queue 1 is not completely transferred. Step S907: The execution unit 2 acquires the same message a from the input queue.

Step S908: The execution unit 2 also starts to write a message acquisition record to the message record table, that is, adding the execution unit 2 to the two items of the "reception time" and the "execution unit ID" of the entry of the message a to obtain the message a. Time and ID of execution unit 2.

Step S909: Since the value of the "commit time" attribute in the entry of the message a has been successfully written in the non-empty value in step S904 in the message record table at this time, the write message of the unit 2 is executed in step S908. The write operation to get the record does not satisfy the write condition, and the write failure is determined.

Step S910: After obtaining the write failure of the message processing record, the execution unit 2 obtains the attribute values in the entry of the message a from the message record table, and obtains the attribute value of the temporary queue address, which is the address of the temporary queue 1 in this example. .

Step S911: The execution unit 2 transfers all the messages in the temporary queue 1 to the output queue according to the address of the temporary queue 1.

Step S912: After the execution unit 2 completely transfers the messages in the temporary queue 2, write a message completion message to the SDB for the message a, that is, write the execution unit to the "Completion Time" attribute in the entry corresponding to the message a. 2 Transfer the time in the temporary queue 2.

Step S913: The execution unit 2 deletes the processed message a from the input queue.

It can be seen that after an execution unit completes processing for one message, if a failure occurs in the process of transferring the processing result in the temporary queue to the output queue, the message will be re-since because the SQS message processing timeout replay mechanism Appears in the input queue. Then another execution unit will obtain the message, but in fact, the execution unit that previously obtained the message has processed the message, but the processing result in the temporary queue has not been completely transferred, but due to the message in the embodiment of the present invention Obtaining the write condition limit of the record, so that the execution unit that obtains the message later does not repeat the process again, but acquires the address of the temporary queue of the failed execution unit, and the message that has not been transferred in the temporary queue Continue all transfer to the output queue. This not only ensures the continued completion of the service, but also avoids the repeated execution of the processing program and shortens the time; further, if the limitation of the message acquisition record writing condition in the embodiment of the present invention is not adopted, the subsequent execution unit 2 will not only Repeatedly executing the business processing program for the same message a, and will continue to store the output new message in its corresponding temporary queue 2, and further transfer all the messages in the temporary queue 2 to the output queue, but actually, before The same processing result of the item in the temporary queue 1 has been transferred to the input by the execution unit 1. Out of the queue, so this will cause problems with duplicate processing results for the same message output.

An embodiment of the present invention further provides an execution unit of a distributed operation system, where the execution unit is configured to process each pending message taken from an input queue in a service step in which it is located, as shown in FIG. The execution unit includes:

The service processing unit 1001 is configured to generate, according to a to-be-processed message obtained from the input queue, a service processing program that runs the service step where the execution unit is located, and generate at least one service processing result message;

The processing result saving unit 1002 is configured to store the generated service processing result message in a temporary queue corresponding to the execution unit;

The first processing result is stored in the judging unit 1003, and is configured to determine whether the other execution unit has not yet stored the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, if not, The temporary queue corresponding to the current execution unit is used as the current queue J|J, and if so, the trigger processing completion determining unit 1004 operates;

The processing completion determining unit 1004 is configured to determine whether the to-be-processed message is completely processed by other execution units, and if the information is not completely processed, the pending queue of the other execution unit is already stored in the pending message. The temporary queue for processing the obtained service processing result message is used as the current temporary queue;

The processing result transfer unit 1005 is configured to transfer all the service processing result messages in the current temporary queue to the output queue, and the to-be-processed message is completely processed.

It can be seen that, in the execution unit provided by the embodiment of the present invention, before the processing result transfer unit transfers the service processing result message obtained by processing the to-be-processed message to the output queue, the processing completion judgment unit first determines whether the device still returns No other execution unit stores the service processing result message obtained by processing the to-be-processed message into the corresponding temporary queue. If the judgment result is not, the processing result transfer unit in the execution unit outputs the execution unit. The business process result message is transferred to the output queue; otherwise, the process result transfer will transfer the business process result message that has been processed by other execution units to the output queue. This ensures that for the same pending message in the input queue, only one execution unit processes the output of the business processing result message. It is sent to the output queue to avoid outputting duplicate service processing result messages to the output queue for the same pending message.

Similar to the discussion in the previous method embodiment, the execution unit provided in the embodiment of the above execution unit ensures that duplicate service processing result messages are not output to the output queue for the same pending message, but may also occur. Such a situation: that is, before an execution unit executes a business process for a piece of pending message, there may already be other execution units (such as execution unit B) that perform the same business process on the message to be processed, and have already The output service processing result message is stored in the corresponding temporary queue, but since the execution unit A transfers the service processing result message in the temporary queue to the output queue, it is determined that the execution unit B has processed the message. The output message has been stored in the temporary queue of execution unit B, but execution unit A still executes the business processing program for the same pending message, which actually causes waste of processing resources.

In order to avoid such a waste phenomenon, an optimization method is also proposed in the embodiment of the present invention. As shown in FIG. 11, the second processing result is added to the judgment unit on the basis of the above embodiment. 1006. The second processing result storage determining unit 1006 is configured to determine, before the service processing unit works, whether another execution unit stores the service processing result message outputted by the processing of the to-be-processed message into its corresponding temporary queue. If not, the service processing unit 1001 is notified to perform the operation; if so, the notification processing completion unit 1004 performs the operation.

The structure and function of the unit shown in FIG. 11 which is the same as the reference numeral in FIG. 10 are the same. Reference may be made to the description of the relevant unit in FIG. 10 in the above embodiment, and details are not described herein again. In a specific embodiment of the present invention, a message record table is created for each service step, and an entry is created in the message record table for each pending message in the input queue of the corresponding service step. The execution unit shown in Figs. 10 and 11 further includes means for writing various records in the message record table, and the corresponding judgment condition is also determined based on whether or not the write record is successful.

Specifically, the execution unit may further include a processing record writing unit, configured to: after the processing result saving unit stores all the service processing result messages generated by processing the processing message into the temporary queue corresponding to the current execution unit, To the message record table corresponding to the pending message A message processing record is written in the entry, and the condition for successful writing is that the message processing record in the entry is empty. At this time, the first processing result storage unit 1003 may include: a processing record writing determining unit, configured to determine whether the processing record writing unit writes the message processing record to the message recording table for the to-be-processed message, if If the write succeeds, it is determined that no other execution unit stores the service processing result message outputted by the processing of the pending message into the temporary queue corresponding to the other execution unit. If the write fails, it is determined that other execution units will The service processing result message outputted by the processing of the to-be-processed message is stored in a temporary queue corresponding to the other execution units.

In addition, for the execution unit shown in FIG. 11, the execution unit may further include an acquisition record writing unit, configured to: after the execution unit acquires a to-be-processed message from the input queue, to the message record table The message acquisition record is written in the entry corresponding to the processing message, and the condition for successful writing is that the message processing record in the entry is empty.

At this time, the second processing result storage unit 1006 includes: an acquisition record writing determination unit, configured to determine whether the acquisition record writing unit writes a message to the message record table for the to-be-processed message to obtain whether the record is successful.

In addition, the execution unit shown in FIG. 10 and FIG. 11 may further include a completion record writing unit, configured to: after the processing result transfer unit transfers all the service processing result messages in the current temporary queue to the output queue, Writing a message completion record in an entry corresponding to the to-be-processed message in the message record table;

The processing completion judging unit includes: a completion record writing judging unit, configured to determine whether a message completion record has been written in an entry corresponding to the to-be-processed message, and if it has been written, determining that the to-be-processed message has been executed by another The unit is completely processed, and if it has not been written, it is determined that the pending message has not been completely processed by other execution units.

Further, another embodiment of the present invention provides a distributed operating system including an execution unit of the above-described embodiments and a storage unit that stores a message recording table.

It should be noted that, if a to-be-processed unit has been completely processed by an execution unit, and the service processing result is output to the output queue, the to-be-processed message may be deleted. Based on such considerations, the execution in the embodiment of the present invention is performed. The unit may further include: a deleting unit, configured to delete the to-be-processed message from the input queue after the processing result forwarding unit processes the to-be-processed message.

In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and parties The law can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment. In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The functions may be stored in a computer readable storage medium, including instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all of the methods described in various embodiments of the present invention or Part of the steps. The foregoing storage medium includes: a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, which can store program codes. .

The above disclosure is only a few specific embodiments of the present invention, but the present invention is not limited thereto, and any changes that can be made by those skilled in the art should fall within the protection scope of the present invention.

Claims

Rights request

A service processing method for a distributed operating system, characterized in that an execution unit in each service step performs the following operations for each pending message retrieved from an input queue of the service step:

The current execution unit, for a message to be processed obtained from the input queue, runs a service processing program of the business step where the execution unit is located, and generates at least one service processing result message;

The method according to claim 1, wherein the method further comprises: before the current execution unit runs the service processing program for the acquired to-be-processed message, performing the following steps: determining whether there are no other execution units yet The service processing result message outputted by the processing of the to-be-processed message is stored in the temporary queue corresponding to the other execution unit, and if not, the execution of the service processing program for the acquired pending message is continued; if yes, Returning to the execution execution unit, all the business processing result messages in the current temporary queue are transferred to the output queue, and the processing of the pending processing message is completely processed by the current execution unit.

The method according to claim 1 or 2, wherein the method further comprises: creating a message record table for each service step, where the message record table is in an input queue corresponding to the service step Create an entry for each pending message;

All the business processing result messages generated by the current execution unit to process the to-be-processed message After being stored in the temporary queue corresponding to the current execution unit, the current execution unit writes a message processing record to the entry corresponding to the to-be-processed message in the message record table, and the condition for successful writing is the message in the entry. Processing record is empty;

The determining whether the other execution unit has not processed the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit includes:

Determining whether the current execution unit writes the message processing record to the message record table for the pending message is successful. If the writing is successful, the determination is not. If the writing fails, the determination is yes.

The method according to claim 2, wherein the method further comprises: creating a message record table for each service step, and corresponding service steps in the message record table. Create an entry for each pending message in the input queue;

After the current execution unit stores the service processing result message generated by processing the to-be-processed message in the temporary queue corresponding to the current execution unit, the current execution unit corresponds to the to-be-processed message in the message record table. The message processing record is written in the entry, and the condition for successful writing is that the message processing record in the entry is empty;

After the current execution unit obtains a pending message from the input queue, the message acquisition record is written into the entry corresponding to the to-be-processed message in the message record table, and the condition for successful writing is that the message processing record in the entry is empty. ;

The determining whether the other execution unit has not yet stored the service processing result outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit includes:

Determining whether the current execution unit writes a message to the message record table for the pending message to obtain whether the record is successful. If the write is successful, the determination is not. If the write fails, the determination is made.

The method according to claim 3 or 4, wherein the current execution unit includes, in the message processing record written in the entry corresponding to the to-be-processed message in the message record table, the current execution The temporary queue address corresponding to the unit;

The temporary queue of the service processing result message that has been processed in the temporary queue of the other execution unit and has been processed into the pending processing message is used as the current temporary queue.

Obtaining, in the message processing record that has been successfully written, from the entry corresponding to the to-be-processed message Temporary queue address;

The temporary queue that is located by the obtained temporary queue address is used as the current temporary queue.

The method according to claim 5, wherein the current execution unit further includes: the current execution unit in the message processing record written in the entry corresponding to the to-be-processed message in the message record table All the service processing result messages generated by processing the to-be-processed message are all stored in the commit queue corresponding to the current execution unit;

The blank of the message processing record in the corresponding entry of the to-be-processed message means that the submission time in the entry is empty.

After the current execution unit transfers all the service processing result messages in the current temporary queue to the output queue, the current execution unit also writes a message completion record to the entry corresponding to the to-be-processed message in the message record table;

Determining whether the to-be-processed message has been completely processed by the other execution unit comprises: determining whether a message completion record has been written in the entry corresponding to the to-be-processed message, and if it has been written, determining that the pending message has been executed by another The unit is completely processed, and if it has not been written, it is determined that the pending message has not been completely processed by other execution units.

The method according to claim 7, wherein the current execution unit records the message completion record written in the entry corresponding to the to-be-processed message in the message record table, and the current execution unit includes the current temporary queue. The service processing result information in all is transferred to the completion time of the output queue; the message completion record written in the entry corresponding to the to-be-processed message means that the completion time in the entry is not empty.

The method according to any one of claims 1 to 8, wherein the method further comprises:

In the case of the complete processing in step D, the current execution unit takes the next pending message from the input queue and returns to the execution step.

After the to-be-processed message is completely processed by the current execution unit in step E, the current execution unit deletes the to-be-processed message from the input queue.

An execution unit of a distributed operating system, wherein the execution unit is configured to process, in a business step in which it is located, each pending message retrieved from an input queue, the execution unit comprising:

a service processing unit, configured to: for a pending message obtained from the input queue, run a service processing program of the service step where the execution unit is located, generate at least one service processing result message; and process a result saving unit, which is to be generated The service processing result message is stored in a temporary queue corresponding to the execution unit;

The first processing result is stored in the judging unit, and is configured to determine whether the other execution unit has stored the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, if not, The temporary queue corresponding to the current execution unit is used as the current queue, and if so, the processing completion judgment unit is triggered to work;

The execution unit according to claim 11, wherein the execution unit further comprises: a second processing result storage determining unit, configured to determine whether there are other execution units before the service processing unit works Transmitting the service processing result message outputted by the processing of the to-be-processed message into its corresponding temporary queue. If not, notifying the service processing unit to perform the work; if yes, notifying the processing completion determining unit to work .

13. An execution unit according to claim 11 or 12, characterized in that, for each service Each of the steps creates a message record table in which an entry is created for each pending message in the input queue of the corresponding service step;

The execution unit further includes: a processing record writing unit, configured to: after the processing result saving unit stores all the service processing result messages generated by processing the processing message into the temporary queue corresponding to the current execution unit, to the message The message processing record is written in the entry corresponding to the to-be-processed message in the record table, and the condition for successful writing is that the message processing record in the entry is empty;

The first processing result storage determining unit includes: a processing record writing determining unit, configured to determine whether the processing record writing unit writes a message processing record to the message recording table for the to-be-processed message, if the writing is successful, if the writing is successful If it is successful, it is determined that no other execution unit stores the service processing result message outputted by the processing of the to-be-processed message into the temporary queue corresponding to the other execution unit, and if the write fails, it is determined that other execution units will process the The service processing result message outputted by the to-be-processed message is stored in a temporary queue corresponding to the other execution units.

14. The execution unit according to claim 12, wherein a message record table is created for each service step, and each message to be processed in the input queue of the corresponding service step is created in the message record table. An entry;

The execution unit also includes:

a processing record writing unit, configured to: after the processing result saving unit stores all the service processing result messages generated by processing the processing message into the temporary queue corresponding to the current execution unit, to the message record table The message processing record is written in the entry corresponding to the processing message, and the condition for successful writing is that the message processing record in the entry is empty;

Obtaining a record writing unit, configured to: after the execution unit acquires a to-be-processed message from the input queue, write a message acquisition record to an entry corresponding to the to-be-processed message in the message record table, and the condition for successful writing is The message processing record in the entry is empty;

The second processing result storage determining unit includes: an obtaining record writing determining unit, configured to determine whether the acquiring record writing unit successfully writes a record to the message record table for the to-be-processed message.

15. An execution unit according to claim 11 or 12, characterized in that for each service Each of the steps creates a message record table in which an entry is created for each pending message in the input queue of the corresponding service step;

The execution unit also includes:

Completing the record writing unit, configured to write a message to the entry corresponding to the to-be-processed message in the message record table after the processing result transfer unit transfers all the service processing result messages in the current temporary queue to the output queue. Record

The processing completion determining unit includes:

Completing a record write judging unit, configured to determine whether a message completion record has been written in an entry corresponding to the to-be-processed message, and if it has been written, determining that the to-be-processed message has been completely processed by other execution units, if not yet Write, then determine that the pending message has not been completely processed by other execution units.

The execution unit according to any one of claims 11 to 15, further comprising: a deleting unit, configured to: after the processing result transfer unit processes the to-be-processed message, the to-be-processed message Deleted in the input queue.

A distributed operating system, comprising: a plurality of execution units as recited in claims 11 to 16 and a storage unit storing a message record table.