WO2019223596A1 - Method, device, and apparatus for event processing, and storage medium - Google Patents

Abstract

A method, device, and equipment for event processing, and a storage medium, related to the technical field of big data. When an event is generated in a distributed system, the event type of the event is acquired (201); a target event queue is determined among multiple event queues (202); the event is enqueued to the target event queue (203); when the event is dequeued from the target event queue, the event is processed (204). By designing multiple event types and multiple event queues, events are enqueued to corresponding event queues on the basis of the event type of each, the number of event queues is increased, thus increasing the total capacity of the event queues, increasing the capacity of the distributed system in buffering events, and increasing the stability and availability of the distributed system.

Description

Event processing method, device, equipment and storage medium

This application claims priority from a Chinese patent application filed on May 25, 2018 with an application number of 201810545759.6 and an invention name of "Event Processing Method, Apparatus, Equipment, and Storage Medium", the entire contents of which are incorporated herein by reference. .

Technical field

The present application relates to the field of big data technology, and in particular, to an event processing method, device, device, and storage medium.

Background technique

A distributed system is a system composed of a group of node devices that communicate and coordinate work through the network to complete common tasks. At each stage of task execution, the distributed system will generate corresponding events due to the process of executing the task. For example, during the task of performing statistics on license plate data, the distributed system will generate an event of writing license plate data to the storage system. Distributed systems need to process the events that are generated in order to complete the task.

Take the distributed system running based on the Spark architecture as an example. The Spark architecture includes a client (English: client) node. The client node can include an event processing module for processing events. Based on the Spark architecture, the client node creates an event during initialization. Queues and event queues are used to buffer events sent to the event processing module. During the processing of tasks, whenever any event occurs, the client node will list the events into the event queue. At the head of the team, the client node dequeues the event from the event queue and sends the event to the event processing module. Through the event processing module, the event can be processed.

In the process of implementing the present invention, the inventors found that the related technology has at least the following problems:

The capacity of a single event queue is small. Once the events in this event queue reach the capacity limit, new events cannot be accommodated, and the distributed system cannot continue to process new events, which affects the processing performance of the distributed system.

Summary of the Invention

The embodiments of the present application provide an event processing method, device, device, and storage medium, which can solve the technical problem that the capacity of a single event queue in the related technology is limited, resulting in low processing performance of the distributed system. The technical solution is as follows:

In one aspect, an event processing method is provided. The method includes:

When an event is generated in a distributed system, obtaining the event type of the event;

Determining a target event queue from a plurality of event queues based on the event type, and the plurality of event queues are used to separately cache events of multiple event types;

Enumerating the event into the target event queue;

When the event is dequeued from the target event queue, the event is processed.

In a possible implementation manner, determining the target event queue from a plurality of event queues based on the event type includes:

Query routing information to obtain an event queue identifier corresponding to the event type, where the routing information includes multiple event types and corresponding multiple event queue identifiers;

The event queue corresponding to the event queue identifier is used as the target event queue.

In a possible implementation manner, the depth of any one of the multiple event queues is positively related to the time-consuming duration of processing events of the corresponding event type.

In a possible implementation manner, before the obtaining an event type of the event, the method further includes:

For the multiple event types, multiple event queues are generated.

In a possible implementation manner, the method further includes:

Sending events in the event queue to an event processing module concurrently through multiple threads for any event queue in the multiple event queues;

Process the event through the event processing module;

In a possible implementation manner, the number of threads corresponding to any of the event queues in the multiple event queues is positively related to the time consumed for processing events of the corresponding event type.

The processing the event includes:

Matching the event with at least one sub-event type under the event type to obtain the sub-event type matched by the event;

Sending the event to an event processing module corresponding to the sub-event type;

The event processing module processes the event.

In a possible implementation manner, the event processing module corresponding to any one of the multiple event types includes at least two.

In a possible implementation manner, the number of event processing modules corresponding to any one of the multiple event types is positively related to the time-consuming duration of processing events of the corresponding event type.

In a possible implementation manner, the multiple event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, and an event queue corresponding to a system file event type. , At least two of the event queues corresponding to the job event type and the event queues corresponding to other event types.

In another aspect, an event processing apparatus is provided, where the apparatus includes:

An acquisition module, configured to acquire an event type of the event when an event is generated in the distributed system;

A determining module, configured to determine a target event queue from a plurality of event queues based on the event type, and the plurality of event queues are respectively used to buffer events of multiple event types;

Enqueuing module, configured to enqueue the event into the target event queue;

An event processing module is configured to process the event when the event is dequeued from the target event queue.

In a possible implementation manner, the determining module includes:

A query submodule, configured to query routing information to obtain an event queue identifier corresponding to the event type, where the routing information includes multiple event types and corresponding multiple event queue identifiers;

A determining submodule is configured to use an event queue corresponding to the event queue identifier as the target event queue.

In a possible implementation manner, the depth of any one of the multiple event queues is positively related to the time-consuming duration of processing events of the corresponding event type.

In a possible implementation manner, the apparatus further includes:

A generating module is configured to generate multiple event queues for the multiple event types.

In a possible implementation manner, the apparatus further includes:

The sending module is configured to send an event in the event queue to the event processing module concurrently through multiple threads for any event queue in the multiple event queues.

In a possible implementation manner, the number of threads corresponding to any of the event queues in the multiple event queues is positively related to the time consumed for processing events of the corresponding event type.

In a possible implementation manner, the apparatus further includes:

A matching module, configured to match the event with at least one sub-event type under the event type to obtain the sub-event type matched by the event;

The sending module is configured to send the event to an event processing module corresponding to the sub-event type.

In a possible implementation manner, the event processing module corresponding to any one of the multiple event types includes at least two.

In a possible implementation manner, the number of event processing modules corresponding to any one of the multiple event types is positively related to the time-consuming duration of processing events of the corresponding event type.

In a possible implementation manner, the multiple event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, and an event queue corresponding to a system file event type. , At least two of the event queues corresponding to the job event type and the event queues corresponding to other event types.

In another aspect, a computer device is provided. The computer device includes a processor and a memory, and the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the foregoing event processing method.

In another aspect, a computer-readable storage medium is provided. The storage medium stores at least one instruction, and the at least one instruction is executed by a processor to implement the foregoing event processing method.

In another aspect, a computer program product containing instructions is provided, which when run on a computer device, enables the computer device to implement the event processing method described above.

In another aspect, a chip is provided. The chip includes a processor and / or program instructions. When the chip is running, the event processing method is implemented.

The beneficial effects brought by the technical solutions provided in the embodiments of the present application include at least:

The method, device, device and storage medium provided in the embodiments of the present application introduce a multi-queue event caching mechanism for a distributed system. By designing multiple event types and multiple event queues, each event is separately entered according to the event type. Listing to the corresponding event queues increases the number of event queues, thereby increasing the total capacity of the event queues, further improving the ability of distributed systems to cache events, and greatly improving the performance of distributed systems. In particular, in the scenario where the distributed system faces high concurrent access, it can meet the needs of the distributed system to cache a large number of events. In addition, the capacity of the event queue is expanded, and a large number of events can be cached through multiple event queues, which prevents the frequent loss of events caused by insufficient event queue capacity, thereby improving the stability and availability of the distributed system. At the same time, different event queues are used to cache events of different event types, so that a large number of events in the distributed system can be cached separately. The processing of different types of events will not interfere with each other, which improves the processing efficiency of the entire distributed system.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are just some embodiments of the application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without paying creative labor.

FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application; FIG.

2 is a flowchart of an event processing method according to an embodiment of the present application;

3 is a flowchart of an event processing method according to an embodiment of the present application;

4 is a schematic diagram of an event processing method according to an embodiment of the present application;

5 is a schematic structural diagram of an event processing apparatus according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed ways

In the following, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a master node 101, at least one slave node 102, and a client node 103.

The master node 101, the at least one slave node 102, and the client node 103 are connected through a network. The master node 101, the at least one slave node 102, and the client node 103 can form a distributed system and work together to complete tasks. For example, the client node 103 can generate a task to be executed, and send the task to be executed to the master node 101. The master node 101 can assign a task to each slave node 102, each slave node 102 can execute the task, and the result of the task processing Sent to the client node 103.

The architecture of the distributed system includes, but is not limited to, various architectures such as a spark architecture, a flink architecture, a mapreduce architecture, and a storm architecture. For example, when the distributed system is running based on the spark architecture, the client node 103 may be a driver (English: Driver) node in the spark architecture, and the master node 101 may be a cluster manager (English: Cluster Manager) in the spark architecture. The node, that is, the master (English: master) node, and the slave node 102 may be a worker (English: Worker) node in the Spark architecture.

The client node 103 may be a computer device, such as a terminal or a server, and may include a personal computer, a notebook computer, a mobile phone, and the like. The master node 101 and at least one slave node 102 may include a server, a terminal, and the like.

FIG. 2 is a flowchart of an event processing method provided by an embodiment of the present application. The method is executed by a computer device. The method includes the following steps:

201. When an event occurs in a distributed system, obtain an event type of the event.

202. Based on the event type, determine a target event queue from a plurality of event queues, where the plurality of event queues are respectively used to buffer events of multiple event types.

203. Queue the event into the target event queue.

204. When the event is dequeued from the target event queue, process the event.

The method provided in the embodiment of the present application introduces a multi-queue event cache mechanism for a distributed system. By designing multiple event types and multiple event queues, each event is listed in the corresponding event queue according to the event type. , Increasing the number of event queues, thereby increasing the total capacity of the event queue, thereby improving the ability of distributed systems to cache events, and greatly improving the performance of distributed systems. In particular, in the scenario where the distributed system faces high concurrent access, it can meet the needs of the distributed system to cache a large number of events. In addition, the capacity of the event queue is expanded, and a large number of events can be cached through multiple event queues, which avoids the frequent loss of events caused by insufficient event queue capacity, thereby improving the stability and availability of the distributed system. At the same time, different event queues are used to cache events of different event types, so that a large number of events in the distributed system can be cached separately. The processing of different types of events will not interfere with each other, which improves the processing efficiency of the entire distributed system.

In a possible implementation manner, determining the target event queue from multiple event queues based on the event type includes:

Query routing information to obtain the event queue identifier corresponding to the event type. The routing information includes multiple event types and corresponding multiple event queue identifiers.

Use the event queue corresponding to the event queue identifier as the target event queue.

In a possible implementation manner, the depth of any one of the plurality of event queues is positively related to the time consuming time for processing events of the corresponding event type.

In a possible implementation manner, before obtaining the event type of the event, the method further includes:

Generate multiple event queues for the multiple event types.

In a possible implementation manner, the method further includes:

For any event queue in the multiple event queues, multiple events are used to concurrently send events in the event queue to the event processing module;

Process the event through the event processing module;

In a possible implementation manner, the number of threads corresponding to any of the event queues in the multiple event queues is positively related to the time consumed for processing events of the corresponding event type.

In a possible implementation manner, the event processing includes:

Matching the event with at least one sub-event type under the event type to obtain the sub-event type matched by the event;

Sending the event to an event processing module corresponding to the sub-event type;

The event processing module processes the event.

In a possible implementation manner, the event processing module corresponding to any one of the multiple event types includes at least two.

In a possible implementation manner, the number of event processing modules corresponding to any one of the multiple event types is positively related to the time-consuming time for processing events of the corresponding event type.

In a possible implementation manner, the multiple event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, an event queue corresponding to a system file event type, At least two of the event queues corresponding to the job event type and the event queues corresponding to other event types.

FIG. 3 is a flowchart of an event processing method provided by an embodiment of the present application. The method is executed by a computer device, and the computer device may be a node device where an event queue is located in a distributed system. For example, in the Spark architecture, the The computer device may be a driver node, that is, a client node. The method includes:

301. A computer device generates multiple event queues for multiple event types.

In this embodiment, a multi-queue event cache mechanism is designed, and multiple event queues are generated so that events of different event types can be cached through multiple event queues, respectively.

Specifically, the computer device can obtain multiple event types. For each of the multiple event types, the computer device can generate a corresponding event queue for the event type, thereby obtaining multiple event queues. Among them, each event queue is used to cache events of a corresponding event type.

Multiple event types: The division of event types can be determined according to the business needs of the distributed system. In a possible implementation, multiple event types can include heartbeat event types, resource monitoring event types, resource application event types, system file events At least two of the type, job event type, and other event types.

The following types of events are described through (1) to (6):

(1) Heartbeat event type: includes various heartbeat events, for example, it can include the heartbeat event between the master node and each slave node, the heartbeat event between the master service and each slave service, the client node and each slave node. Heartbeat events, heartbeat events between client nodes and master nodes, etc.

(2) Resource monitoring event types: including various events that obtain information on the use of resources in a distributed system. The resources can include CPU (Central Processing Unit), memory, and disk IO (Input / Output, input / output). ), Network bandwidth, etc.

(3) Types of resource application events: Including various events for applying for resources and events for recycling resources, for example, when a job is submitted to a distributed system, events that apply for resources such as CPU and memory, trigger the system to perform gc (Garbage Collection, garbage (Recycling) event, and an event of reclaiming the resource requested by the task after the task execution is completed.

(4) System file event type: includes various events that interact with the storage system. The interaction data includes writing data to the storage system and reading data from the storage system. The storage system can include local storage, hdfs, and databases. , Hard disk, cloud storage, etc., this data can include log logs.

(5) Job event type: Includes events where a client submits a job to a distributed system. When a client submits a job to a distributed system, the distributed system will split the job into multiple job phases, and then each job phase Split into multiple tasks, for example, DAG (Directed Acyclic Graph, Directed Acyclic Graph), Job (Job), Stage (Job Stage), Task (Task) in the Spark architecture are merged into this type of event.

(6) Other event types: All events that do not belong to the above event categories can be merged into other event types.

In combination with the event types described in (1) to (6) above, the multiple event queues generated by the computer device may include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, and an event queue corresponding to a resource application event type. , At least two of the event queue corresponding to the system file event type, the event queue corresponding to the job event type, and the event queue corresponding to other event types. The event queue corresponding to the heartbeat event type is used to buffer events belonging to the heartbeat event type. Resource monitoring The event queue corresponding to the event type is used to cache events belonging to the resource monitoring type, the event queue corresponding to the resource application event type is used to cache events belonging to the resource application event type, and the event queue corresponding to the system file event type is used to cache system file events Type of event, the event queue corresponding to the job event type is used to cache events belonging to the job event type, and the event queues corresponding to other event types are used to cache events belonging to the job event type.

Exemplarily, referring to FIG. 4, the event queue can be expressed as eventQueue, and the computer device can generate 6 event queues for 6 event types, which are in turn eventQueue1, eventQueue2, ..., eventQueue6, where eventQueue1 is the cache heartbeat event type corresponding Event queue, eventQueue2 is the event queue corresponding to the resource monitoring event type, and so on.

In this step, multiple event queues are generated for multiple event types. On the one hand, the number of event queues is increased by adding event queues, so the total capacity of the event queue is increased, and the distributed system cache is also improved. The ability of events further improves the performance and scalability of distributed systems. In particular, in the scenario where the distributed system faces high concurrent access, it can meet the need for the distributed system to cache a large number of events, and improve the processing performance of the distributed computing system. On the other hand, by setting multiple event queues to cache events, the probability of event loss is greatly reduced, and the situation of frequent loss of events by distributed systems is avoided, which also prevents the system from being unstable due to lost resource cleanup events. The hidden dangers of unavailability have improved the stability and availability of the distributed system. On the other hand, by generating different event queues to cache events of different event types, a large number of events in the distributed system can be cached separately. Each event queue is dedicated to cache events of the corresponding event type without paying attention to other event types. Events, alleviating the storage pressure of a single event queue.

Optionally, the computer device may obtain the depths of multiple event queues and generate multiple event queues according to the depths of the multiple event queues, so that the capacity of each event queue can meet business requirements. The depth of the event queue is used to indicate the number of events that the event queue can hold. For example, the depth of the event queue can be equal to the number of events that the event queue can hold. For another example, the depth of the event queue can be equal to the event that the event queue can hold. The ratio between the number of thresholds and a threshold coefficient, which may be 80%, 60%, or the like.

Regarding the depth of each event queue, optionally, the depth of each event queue in the multiple event queues is positively related to the time-consuming duration of processing events of the corresponding event type. Specifically, the depth of each event queue can be designed in conjunction with the time-consuming process of events. The more time-consuming it takes to process a certain type of event, the deeper the corresponding event queue, and the more events it can cache. More, which improves the ability to cache such events. Similarly, the faster the processing of a certain type of event, the shallower the corresponding event queue.

In a possible implementation, for each event type of multiple event types, the depth of the event queue corresponding to the event type can be configured in advance according to the time spent processing the event type. If the event of the event type is processed, The time-consuming duration of the event type is longer, and the depth of the event queue of the event type can be configured to be larger. If the time-consuming duration of processing the event of the event type is shorter, the depth of the event queue can be configured to be smaller. In this way, the computer device can obtain the depth configured for each event queue, and after generating the event queue according to the configured depth of the event queue, it can achieve the effect that the depth of the event queue is positively related to the time-consuming time of processing the event of the corresponding event type.

For example, the time-consuming time for processing events of the heartbeat event type is usually short, you can set the depth of the event queue of the heartbeat event type is small, and the time-consuming time for processing events of the job event type is usually long, you can set the job The event queue has a greater depth.

It should be noted that the above is only described by taking the positive correlation between the depth of the event queue and the processing time of the event as an example. Optionally, the depth of each event queue may also be a default value or an empirical value. This is not limited.

302. When an event is generated in the distributed system, the computer device obtains an event type of the event.

Distributed systems can generate various events during operation. In an exemplary scenario, during the execution of a task, the distributed system triggers a corresponding event due to the execution of the task. For example, in the Spark architecture, when a client submits a job, the Driver node will establish a connection with the Cluster Manager node, register with the Cluster Manager node and apply for resources. For another example, during the execution of tasks, each Worker node can report to the Driver node. Send the heartbeat. For example, after the Driver node gets the job, it can build a DAG graph, decompose the DAG graph into multiple job phases, and decompose each job phase into multiple tasks. Of course, the distributed system can also generate other events in other scenarios. This embodiment does not limit the scenarios that generate events and the specific types of events.

When an event is generated, the computer device can obtain the event type of the event. Specifically, multiple event types may be pre-configured. When an event is generated, the computer device may obtain an event type that matches the event from the pre-configured multiple event types.

For example, a correspondence relationship between an event type and an event name may be set in advance, and each event type corresponds to at least one event name. After an event is generated, the computer device may obtain the name of the event, query the correspondence relationship, and obtain the event. The name of the event type.

303. The computer device determines a target event queue from a plurality of event queues based on the event type.

The computer device can establish the correspondence between the event type and the event queue in advance by means of event routing. After the event type of the event is determined, the event can be determined from multiple event queues based on the event type and the pre-established correspondence. The event queue corresponding to the type uses the event queue corresponding to the event type as the target event queue, so that the generated events are listed in the target event queue.

In a possible implementation, the correspondence between the event type and the event queue may be indicated by routing information. In conjunction with routing information, for the specific process of determining the target event queue, the computer device can use the event type as an index, query the routing information, obtain the event queue identifier corresponding to the event type, and use the event queue corresponding to the event queue identifier as the target event queue.

The routing information is used to indicate the correspondence between the event type and the event queue. The routing information includes multiple event types and corresponding multiple event queue identifiers. The event queue identifier is used to identify the corresponding event queue. Name, number, etc. With reference to the six exemplary event types in step 301, the routing information can be shown in Table 1 below:

Table 1

Event type	Event queue ID
Heartbeat event type	eventQueue1
Resource monitoring event type	eventQueue2
Resource Request Event Type	eventQueue3
System file event type	eventQueue4
Job event type	eventQueue5
Other event types	eventQueue6

304. The computer device queues the event into the target event queue.

Enqueue refers to sending an event to a queue, that is, inserting an event into the queue, so that the event is queued in the queue, and the event is cached. Among them, the event queue can be a FIFO (First Input First Output) queue, and the entry queue can be the event insertion at the end of the event queue.

After the computer device determines the target event queue of the event, it can enqueue the event to the target event queue corresponding to the event type, that is, send the event to the target event queue, that is, insert the event at the end of the target event queue. After that, the event will Queue in the target event queue. When all events that precede the event are dequeued from the target event queue, the event will be queued at the head of the target event queue to be dequeued.

Optionally, in combination with the above steps 303 and 304, the function of event routing can be implemented by querying routing information and enqueuing events to an event queue, that is, each event generated can be routed to a corresponding event queue to achieve The effect of events listed by type. In a possible embodiment, the foregoing steps 303 and 304 may be encapsulated into a routing module, and the function of event routing is implemented by the routing module. The computer device may execute the foregoing steps 303 to 304 by running the routing module.

In an exemplary scenario, whenever a heartbeat event is generated by each node device in a distributed system, various heartbeat events may be routed to an event queue corresponding to the heartbeat event type according to the heartbeat event type. When each node device reads and writes data to the storage system, events such as write data events and read events can be routed to the event queue corresponding to the system file event type according to the system file event type, and so on.

In this embodiment, a refined event caching mechanism is provided. All events are unified into an event queue, and various events are routed to their corresponding event queues. By routing different types of events to different event queues, at least the following technical effects can be achieved:

On the one hand, by introducing the process of event routing between event distribution and event queue, each event can be sent to the corresponding event queue according to the event type, which realizes the function of each event being listed separately according to the event type.

On the other hand, in the related art, because a single event queue is used to cache all events, various types of events are prone to cause mutual interference due to the head-of-line blocking effect. In this embodiment, different types of events are queued separately in different event queues, which can avoid the interference of different types of events due to blockage of the headlines, and also avoid the time-consuming processing of certain types of events, which will affect other types of events. Handle progress and improve the efficiency of event queuing.

In an exemplary scenario, if the log event is ranked first in the event queue, and because the log event is time-consuming, the event processing module that processes the log event is currently busy and the log event cannot be dequeued, resulting in the log event in the event queue. Short-lived events, such as heartbeat events, are also blocked in the event queue, unable to dequeue from the event queue, and cannot be sent to the corresponding event processing module, which affects the event processing efficiency of the entire distributed system.

In the embodiment of the present application, different types of events are queued separately. The congestion of the event queue of the log event will not interfere with the event queue of the heartbeat event. Even if the log event is blocked in the event queue corresponding to the log event type, the heartbeat event can be Normally queue and dequeue from the event queue corresponding to the heartbeat event type, thereby improving the event processing efficiency of the entire distributed system.

305. When the event is dequeued from the target event queue, the computer device sends the event to the event processing module.

Over time, each event in the target event queue will move from the end of the team to the head of the team. When the generated event is ranked at the head of the target event queue, the computer device will send the event to the event processing module.

Among them, the event processing module can also be called event handler, listener, etc. The event processing module is used to process events. It can be a virtual program module and can be executed by a thread, object, process or other program execution unit in a computer device. The event processing module encapsulates a method for processing events, and the event processing module can call the encapsulated method to process the event.

Regarding the process of sending an event to the event processing module, the computer device may generate a thread for sending an event to the event processing module, and send the event to the event processing module through the thread. Among them, the thread refers to the execution flow of the program, and is the basic unit for CPU execution. For example, the thread used to send events can be a daemon thread, and the daemon thread can listen to the target event queue, and when the event is dequeued from the target event queue At this time, the daemon thread can obtain the event and send the event to the event processing module.

Optionally, events can be distributed to the event processing module concurrently through multiple threads. Among them, Concurrency refers to a mechanism in which multiple threads execute tasks in turn. For example, for thread A, thread B, and thread C, these three threads execute tasks concurrently, that is, thread A executes the task first, and then thread B executes the task. Task, and then thread C executes the task. Among them, because the time interval between different threads is extremely short, it can be regarded as multiple tasks being executed at the same time. The multi-threaded concurrency mechanism can greatly improve the overall efficiency of task execution.

Combined with the multi-threaded concurrency mechanism, for any event queue in multiple event queues, events in the event queue can be sent to the event processing module concurrently through multiple threads. That is, multiple threads will send events to the event processing module in turn. When a thread sends an event to the event processing module, there is no need to wait for the thread to finish sending. Instead, the next thread continues to send events to the event processing module.

Exemplarily, it is assumed that events are sent through the first thread and the second thread concurrently. For any two adjacent events in the event queue, the previous event is called the first event and the latter event is called the second event. For example, the process of sending these two events through these two threads can include the following steps one to two:

Step 1: When the first event is dequeued from the event queue, an event is sent to the event processing module through the first thread.

Step 2: When the second event is dequeued from the event queue, the second event is sent to the event processing module through the second thread, where the second thread is different from the first thread.

When the first event is dequeued from the event queue, the second event after the first event is moved to the head of the team and dequeued from the event queue. At this time, there is no need to wait for the first thread to finish sending the first event, and it is sufficient to send the second event directly through the second thread.

It should be noted that the number of the multiple threads may be two or more, and the data of the multiple threads is specifically determined according to business requirements, which is not limited in this embodiment.

Distributing events through multiple threads can achieve at least the following technical effects:

In the related art, a single thread is used in a distributed system to serially send events to an event processing module. That is, a thread is fixed to send events in the event queue to the event processing module. When the current event is dequeued from the event queue, the thread must obtain the previous event, send the previous event to the event processing module, and wait for the previous one. After the event is sent, the thread can continue to send the next event, so the efficiency of sending the event is very low.

In the embodiment of the present application, multiple threads can send events concurrently, and multiple threads can send each event in the event queue in turn. The multi-thread mechanism greatly improves the speed of sending events, and the previous one in the event queue The sending process of the event will not block the sending process of the next event, thereby greatly improving the efficiency of sending the event.

Optionally, on the basis of sending events by multiple threads, the number of threads sending events for each event queue may be designed in combination with the time consumption of processing events. Specifically, the number of threads corresponding to any event queue in multiple event queues is positively related to the time spent processing events of the corresponding event type, that is, the more time it takes to process a certain type of event, the event queue is for such events The greater the number of threads that send events, improving the ability to send such events. Similarly, the faster the processing of certain types of events, the fewer the number of threads that send events to the event queue of such events, such as a single thread, thereby saving system resources.

In a possible implementation, for each event type of multiple event types, the number of threads of the event queue corresponding to the event type can be configured in advance according to the time spent processing the event type. If the event type is processed, The event takes a long time, and you can configure more threads of the event type event queue. If the event type of the event type has a shorter time, you can configure the event type event queue thread Less, in this way, the computer device can obtain the number of threads configured for each event queue, and based on the number of threads configured for the event queue, after generating the corresponding threads for each event queue, the number of threads and the processing of the corresponding event type can be achieved. The time-consuming effect of the event is positively related.

For example, it usually takes a short time to process a heartbeat event type event. For a heartbeat event type event queue, you can set the event queue to still send the events of the event queue serially according to a single thread. However, it usually takes a long time to process a job event type event. For a job event type event queue, multiple threads can be set to send events in the event queue concurrently.

In this step, an event queue for time-consuming events can be sent through multiple threads, and an event queue for non-time-consuming events can be sent through a single thread. In this way, the flexibility of the process of sending events through threads is improved, and the ability to send time-consuming events is significantly improved, and the events in the time-consuming event queue can be specifically processed.

Optionally, multiple event processing modules may be introduced to process events concurrently, that is, for each event type of multiple event types, all events of the event type may be collectively processed through multiple event processing modules. Specifically, the event processing module corresponding to any one of the multiple event types may include at least two. During the processing of an event by an event processing module, there is no need to wait for the event processing module to finish processing. The event processing module continues to process events.

Optionally, the number of event processing modules corresponding to any one of the multiple event types is positively related to the time-consuming time for processing the event of the corresponding event type, that is, if the time-consuming time corresponding to the event type is longer, the The greater the number of event processing modules corresponding to the event type, the stronger the distributed system can handle such events. For example, the number of event processing modules of the heartbeat event type is larger, and the number of event processing modules of the job event type is larger. Less in quantity.

In this step, by introducing multiple event processing modules to process events, the degree of concurrency of processing events can be improved, thereby improving the concurrent performance and availability of the distributed system. Further, by combining the time consumption of processing events, the number of event processing modules for each type of event is designed to improve the flexibility of processing events, and at the same time significantly improve the ability to process time-consuming events. Special events are handled at specific times.

Optionally, for the process of determining the event processing module, at least one sub-event type under the event type may be determined, the event is matched with at least one sub-event type under the event type, and the sub-event type matched by the event is obtained, and the The event processing module corresponding to the event type sends an event.

Event types and sub-event types: You can think of event types as large classes and sub-event types as small classes. Sub-event types are more specific and detailed types than the dimensions of event types. They are types that belong to event types. Each event type can include one or more sub-event types. For example, the job event type may include a type of starting a job, a type of ending a job, a type of starting a task, a type of ending a task, and the like.

Regarding the process of determining at least one sub-event type under the event type, all sub-event types can be classified into corresponding event types in advance, and the event type and all sub-event types under the event type are stored on the computer device correspondingly. After the device determines the event type of the event, it can obtain at least one sub-event type under the event type.

Regarding the process of matching the event and the sub-event type under the event type, the event can be matched with each sub-event type in turn. For example, all sub-event types under the event type can be traversed. During the traversal, for the current traversal, To determine the event type that matches the event type. When the event matches the event type, the event type is used as the event type.

Among them, in a possible implementation, the name of the sub-event type can be stored in advance to determine whether the name of the event is the same as the name of the sub-event type. When the name of the event is the same as the name of the sub-event type, the event and the sub-type Event type matches.

Regarding the process of determining the event processing module according to the sub-event type, the correspondence between the sub-event type and the event processing module can be established in advance. After the event-matched sub-event type is obtained, the sub-event type can be determined according to the pre-established correspondence. Corresponding event processing module.

The following describes at least the technical effects that can be achieved by the above process of determining the event processing module:

In related technologies, when an event is generated in a distributed system, all sub-event types need to be used as the traversal range, and the event and all sub-event types must be matched in order to find the matching sub-event type, and then find the appropriate event. Processing module for processing. In this way, the scope of traversal is very large, and it takes a long time to find the matching sub-event type and event processing module, which affects the processing time of the event, and it cannot meet the challenge of increasingly diverse events in the distributed system.

In an exemplary scenario, it is assumed that there are a total of 1000 seed event types in a distributed system. When an event of the 1000th seed event type occurs, it is necessary to traverse from the 1st seed event type to the 1000th seed event type. The seed event types are sequentially matched, and the matching event processing module can be determined only when the 1000th match is performed.

In this embodiment, by dividing a large number of events into multiple event types, matching the event with at least one sub-event type under the event type, narrowing the traversal range from all sub-event types to those under a certain event type. All child event types. That is, it is not necessary to match the event with all sub-event types in order, it is only necessary to match the event with the sub-event type under a certain type, which narrows the scope of traversal, improves the efficiency of matching, and can quickly find the event. Matching event processing module.

In an exemplary scenario, it is assumed that there are a total of 1000 seed event types in a distributed system, which are divided into 10 event types, where the 900th seed event type to the 1000th seed event type are divided into the 10th event type. Then, if an event of the 1000th seed event type occurs, after determining that the event belongs to the 10th event type, there is no need to match the event with the 1st seed event type to the 1000th seed event type, only the event from 900th Match the seed event type to the 1000th seed event type.

306. The computer device processes the event through the event processing module.

After the event processing module receives the event, it can call its own method to process the event and obtain the processing result of the event.

The event cache mechanism in current distributed systems (such as spark) includes the following features:

First, all types of events are cached through a fixed-length event queue.

Therefore, when the number of events in this event queue reaches the upper limit of the capacity of the event queue, if a new event occurs, the new event cannot be listed in the event queue, resulting in the loss of the new event and the failure of the task execution.

Further, if the missing new event is an event that triggers resource reclamation and other types of events such as gc, an OOM (OutOfMemoryError, memory leak) mechanism will be triggered, that is, the operating system will kill the process to release memory, affecting the node device where the event queue is located The normal operation of the device may even cause the node device to crash or be paralyzed, resulting in the node device not being able to communicate with other node devices in the distributed system and affecting the operation of the distributed system. That is, the loss of events will affect the stability of the distributed system with a great probability, and easily cause the situation that the distributed system is unavailable.

Second, all events are sent serially to the event processing module through the same thread. After the thread sends the previous event, the thread can continue to send the next event.

Therefore, each event has to wait for all events before the event in the event queue to finish sending before it can be sent. It can be seen that the efficiency of sending events is extremely low. In addition, if a certain type of event processing takes time and congestion occurs in the event queue, it will affect the dequeue of other types of events in the event queue due to the blocking effect of the head. The efficiency with which the system processes events.

Third, before an event enters the event processing module, the event and all sub-event types need to be matched one by one in order to find a matching event processing module for processing. This matching method requires a large range of traversal. Not only does it affect the efficiency of the distributed system in processing events, it may also cause the distributed system to crash or paralyze.

The embodiments of the present application solve the above-mentioned technical problems, and propose an optimized scheme for processing events. In this solution, by reclassifying events in a distributed system and creating different event queues for different event types, and introducing an event routing method between events and event queues, the event distribution and processing are improved. effectiveness. Furthermore, it not only solves the problem of inefficient event queue capacity in the current distributed system, or the low efficiency of the entire system event processing caused by a time-consuming operation, but also the problem of insufficient performance caused by a single thread sending an event. The operating system OOM caused by the problem, which in turn causes communication problems between node devices and system crashes or paralysis. At the same time, through the mechanism of multi-thread and multi-event processing modules, the concurrency and stability of the distributed system is further improved.

The method provided in the embodiment of the present application introduces a multi-queue event cache mechanism for a distributed system. By designing multiple event types and multiple event queues, each event is listed in the corresponding event queue according to the event type. , Increasing the number of event queues, thereby increasing the total capacity of the event queue, thereby improving the ability of distributed systems to cache events, and greatly improving the performance of distributed systems. In particular, in the scenario where the distributed system faces high concurrent access, it can meet the needs of the distributed system to cache a large number of events. In addition, the capacity of the event queue is expanded, and a large number of events can be cached through multiple event queues, which avoids the frequent loss of events caused by insufficient event queue capacity, thereby improving the stability and availability of the distributed system. At the same time, different event queues are used to cache events of different event types, so that a large number of events in the distributed system can be cached separately. The processing of different types of events will not interfere with each other, which improves the processing efficiency of the entire distributed system.

FIG. 5 is a schematic structural diagram of an event processing apparatus according to an embodiment of the present application. Referring to FIG. 5, the apparatus includes: an obtaining module 501, a determining module 502, an enqueuing module 503, and an event processing module 504.

An obtaining module 501, configured to obtain an event type of an event when an event is generated in the distributed system;

A determining module 502, configured to determine a target event queue from a plurality of event queues based on the event type, and the multiple event queues are respectively used to buffer events of multiple event types;

The enqueuing module 503 is configured to enqueue the event into the target event queue;

An event processing module 504 is configured to process the event when the event is dequeued from the target event queue.

The device provided in the embodiment of the present application introduces a multi-queue event cache mechanism for a distributed system. By designing multiple event types and multiple event queues, each event is individually listed in the corresponding event queue according to the event type. , Increasing the number of event queues, thereby increasing the total capacity of the event queue, thereby improving the ability of distributed systems to cache events, and greatly improving the performance of distributed systems. In particular, in the scenario where the distributed system faces high concurrent access, it can meet the needs of the distributed system to cache a large number of events. In addition, the capacity of the event queue is expanded, and a large number of events can be cached through multiple event queues, which avoids the frequent loss of events caused by insufficient event queue capacity, thereby improving the stability and availability of the distributed system. At the same time, different event queues are used to cache events of different event types, so that a large number of events in the distributed system can be cached separately. The processing of different types of events will not interfere with each other, which improves the processing efficiency of the entire distributed system.

In a possible implementation manner, the determining module 502 includes:

The query submodule is used to query routing information to obtain an event queue identifier corresponding to the event type, and the routing information includes multiple event types and corresponding multiple event queue identifiers;

A determining submodule is configured to use the event queue corresponding to the event queue identifier as the target event queue.

In a possible implementation manner, the depth of any one of the plurality of event queues is positively related to the time consuming time for processing events of the corresponding event type.

In a possible implementation manner, the apparatus further includes:

A generating module is configured to generate multiple event queues for the multiple event types.

In a possible implementation manner, the apparatus further includes:

The sending module is configured to send events in the event queue to the event processing module 504 concurrently through multiple threads for any event queue in the multiple event queues.

In a possible implementation manner, the number of threads corresponding to any of the event queues in the multiple event queues is positively related to the time consumed for processing events of the corresponding event type.

In a possible implementation manner, the apparatus further includes:

A matching module, configured to match the event with at least one sub-event type under the event type to obtain a sub-event type matched by the event;

The sending module is configured to send the event to an event processing module 504 corresponding to the sub-event type.

In a possible implementation manner, the event processing module 504 corresponding to any one of the multiple event types includes at least two.

In a possible implementation manner, the number of the event processing modules 504 corresponding to any one of the multiple event types is positively related to the time consuming time for processing the events of the corresponding event type.

In a possible implementation manner, the multiple event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, an event queue corresponding to a system file event type, At least two of the event queues corresponding to the job event type and the event queues corresponding to other event types.

All the above-mentioned optional technical solutions may be used in any combination to form optional embodiments of the present disclosure, which will not be described in detail here.

It should be noted that the event processing device provided in the foregoing embodiment only uses the division of the foregoing functional modules as an example for processing events. In practical applications, the above functions may be allocated by different functional modules as required. The internal structure of the event processing device is divided into different functional modules to complete all or part of the functions described above. In addition, the event processing device and the event processing method embodiments provided by the foregoing embodiments belong to the same concept. For specific implementation processes, refer to the method embodiments, and details are not described herein again.

FIG. 6 is a schematic structural diagram of a computer device according to an embodiment of the present application. The computer device 600 may have a large difference due to different configurations or performance, and may include one or more processors (central processing units) (CPU) 601. And one or more memories 602, where at least one instruction is stored in the memory 602, and the at least one instruction is loaded and executed by the processor 601 to implement the event processing methods provided by the foregoing method embodiments. Of course, the computer device may also have components such as a wired or wireless network interface and an input-output interface for input and output. The computer device may also include other components for implementing the functions of the device, and details are not described herein.

In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a computer device to complete the event processing method in the foregoing embodiment. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, the present application further provides a computer program product containing instructions, which when executed on a computer device, enables the computer device to implement the event processing method in the foregoing embodiment.

In an exemplary embodiment, the present application further provides a chip that includes a processor and / or program instructions. When the chip runs, the event processing method in the foregoing embodiment is implemented.

Those of ordinary skill in the art may understand that all or part of the steps for implementing the foregoing embodiments may be completed by hardware, or may be instructed by a program to complete related hardware. The program may be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk.

The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present application shall be included in the protection of the present application. Within range.

Claims (22)

1. An event processing method, characterized in that the method includes:

   When an event is generated in a distributed system, obtaining the event type of the event;

   Determining a target event queue from a plurality of event queues based on the event type, and the plurality of event queues are used to separately cache events of multiple event types;

   Enumerating the event into the target event queue;

   When the event is dequeued from the target event queue, the event is processed.
2. The method according to claim 1, wherein determining the target event queue from a plurality of event queues based on the event type comprises:

   Query routing information to obtain an event queue identifier corresponding to the event type, where the routing information includes multiple event types and corresponding multiple event queue identifiers;

   The event queue corresponding to the event queue identifier is used as the target event queue.
3. The method according to claim 1, wherein the depth of any one of the plurality of event queues is positively related to the time consuming time for processing events of the corresponding event type.
4. The method according to claim 1, wherein before the obtaining the event type of the event, the method further comprises:

   For the multiple event types, multiple event queues are generated.
5. The method according to claim 1, further comprising:

   Sending events in the event queue to an event processing module concurrently through multiple threads for any event queue in the multiple event queues;

   The event processing module processes the event.
6. The method according to claim 5, wherein the number of threads corresponding to any one of the plurality of event queues is positively related to the time-consuming time for processing events of the corresponding event type.
7. The method according to claim 1, wherein the processing the event comprises:

   Matching the event with at least one sub-event type under the event type to obtain the sub-event type matched by the event;

   Sending the event to an event processing module corresponding to the sub-event type;

   The event processing module processes the event.
8. The method according to any one of claims 5 to 7, wherein the event processing module corresponding to any one of the multiple event types includes at least two.
9. The method according to claim 8, wherein the number of event processing modules corresponding to any one of the plurality of event types is positively related to the time-consuming time for processing events of the corresponding event type.
10. The method according to claim 1, wherein the plurality of event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, and a system file event type. At least two of the corresponding event queue, the event queue corresponding to the job event type, and the event queue corresponding to other event types.
11. An event processing device, characterized in that the device includes:

    An acquisition module, configured to acquire an event type of the event when an event is generated in the distributed system;

    A determining module, configured to determine a target event queue from a plurality of event queues based on the event type, and the plurality of event queues are respectively used to buffer events of multiple event types;

    Enqueuing module, configured to enqueue the event into the target event queue;

    An event processing module is configured to process the event when the event is dequeued from the target event queue.
12. The apparatus according to claim 11, wherein the determining module comprises:

    A query submodule, configured to query routing information to obtain an event queue identifier corresponding to the event type, where the routing information includes multiple event types and corresponding multiple event queue identifiers;

    A determining submodule is configured to use an event queue corresponding to the event queue identifier as the target event queue.
13. The apparatus according to claim 11, wherein the depth of any one of the plurality of event queues is positively related to the time consuming time for processing events of the corresponding event type.
14. The device according to claim 11, further comprising:

    A generating module is configured to generate multiple event queues for the multiple event types.
15. The device according to claim 11, further comprising:

    The sending module is configured to send an event in the event queue to the event processing module concurrently through multiple threads for any event queue in the multiple event queues.
16. The device according to claim 15, wherein the number of threads corresponding to any one of the plurality of event queues is positively related to the time-consuming time for processing events of the corresponding event type.
17. The device according to claim 11, further comprising:

    A matching module, configured to match the event with at least one sub-event type under the event type to obtain the sub-event type matched by the event;

    The sending module is configured to send the event to an event processing module corresponding to the sub-event type.
18. The apparatus according to any one of claims 14 to 17, wherein the event processing module corresponding to any one of the multiple event types includes at least two.
19. The device according to claim 18, wherein the number of event processing modules corresponding to any one of the plurality of event types is positively related to the time-consuming duration of processing events of the corresponding event type.
20. The device according to claim 11, wherein the plurality of event queues include an event queue corresponding to a heartbeat event type, an event queue corresponding to a resource monitoring event type, an event queue corresponding to a resource application event type, and a system file event type At least two of the corresponding event queue, the event queue corresponding to the job event type, and the event queue corresponding to other event types.
21. A computer device, wherein the computer device includes a processor and a memory, and the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement any one of claims 1-10. Method steps described in item.
22. A computer-readable storage medium, characterized in that at least one instruction is stored in the storage medium, and the at least one instruction is executed by a processor to implement the method steps of any one of claims 1-10.

Images