CN113965584B

CN113965584B - Message processing method, device, apparatus and storage medium

Info

Publication number: CN113965584B
Application number: CN202111571829.3A
Authority: CN
Inventors: 朱超; 倪顺; 夏桂林; 苏治武; 杨瀚清
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-12-21
Filing date: 2021-12-21
Publication date: 2022-05-13
Anticipated expiration: 2041-12-21
Also published as: CN113965584A

Abstract

The embodiment of the application provides a message processing method, message processing equipment, a message processing device and a storage medium. In the embodiment of the application, the attribute information statistics is performed on the messages in the current statistics period through the sliding window, for each attribute information, the first attribute characteristic corresponding to the attribute information in each sliding window in the current statistics period and the second attribute characteristic corresponding to the attribute information in each sliding window in the current statistics period are determined, and the messages in the current statistics period are analyzed according to the first attribute characteristic and the second attribute characteristic, so that the accuracy of abnormal message detection can be improved, the scale of the abnormal messages is determined, message copying among different devices is not required to be designed, system resources can be saved, and message analysis is performed in complex scenes such as large flow, high concurrency, high-performance storage and mixing.

Description

Message processing method, device, apparatus and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a method, a device, and an apparatus for processing a packet, and a storage medium.

Background

Currently, in a C/S architecture mode, a Client (Client) is responsible for interacting with a user, sending a user access request to a Server (Server), and the Server is responsible for data processing and returning a data processing result to the Client. The access request sent by the client to the server may be routed to the storage node by the server, and the storage node provides the data required by the client. For a certain storage node, the Query Per Second (QPS) that can be carried by itself is limited, and if the number of access requests exceeds the upper limit, the stability of the system may be damaged, and even the risk of system avalanche may exist.

For the storage node, the abnormal messages have great potential harm, and the first step of treating the abnormal messages is to find the abnormal messages. At present, the method for discovering an abnormal message is as follows: and performing link replication on the message from the service end, sending the replicated message to a message detection service, and finding an abnormal message based on the message detection service. However, in an application scenario with high concurrency and large traffic, message replication requires significant system resources, which affects system performance, and therefore, in an application scenario with high concurrency and large traffic, a method for discovering an abnormal message is urgently needed.

Disclosure of Invention

Aspects of the present application provide a method, an apparatus, a device, and a storage medium for processing a packet, so as to improve system performance in a high-concurrency and high-traffic application scenario.

The embodiment of the application provides a message processing method, which comprises the following steps: receiving one or more first messages, wherein the first messages have one or more attribute information; aiming at any kind of attribute information, determining a first attribute characteristic of the second message corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the second message corresponding to the attribute information in each sliding period corresponding to each sliding window; the second message is one or more first messages in the current statistical period; and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

An embodiment of the present application further provides a packet processing apparatus, including: the device comprises a receiving module, a first determining module and a second determining module; the receiving module is used for receiving one or more first messages, and the first messages have one or more attribute information; the first determining module is used for determining a first attribute characteristic of the second message corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the second message corresponding to the attribute information in each sliding period corresponding to each sliding window according to any attribute information; the second message is one or more first messages in the current statistical period; and the second determining module is used for determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

An embodiment of the present application further provides a packet processing device, including: a memory and a processor; a memory for storing a computer program; a processor coupled with the memory for executing the computer program for: receiving one or more first messages, wherein the first messages have one or more attribute information; aiming at any kind of attribute information, determining a first attribute characteristic of the second message corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the second message corresponding to the attribute information in each sliding period corresponding to each sliding window; the second message is one or more first messages in the current statistical period; and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

The embodiments of the present application further provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the message processing method provided in the embodiments of the present application.

The embodiment of the present application further provides a computer program product, which includes a computer program/instruction that when executed by a processor, implements the steps in the message processing method provided in the embodiment of the present application.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

in the embodiment of the application, attribute information statistics is performed on the messages in the current statistical period through the sliding window, for each attribute information, a first attribute feature corresponding to the attribute information in each sliding window in the current statistical period and a second attribute feature of the message corresponding to the attribute information in each sliding window in the current statistical period are determined, and the messages in the current statistical period are analyzed according to the first attribute feature and the second attribute feature, so that the accuracy of abnormal message detection can be improved, the scale of the abnormal messages is determined, message copying among different devices is not required, system resources can be saved, and message analysis is performed in complex scenes such as large flow, high concurrency, mixed storage of different high and low performance energy flows and the like.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.

Fig. 1 is a schematic structural diagram of a service system provided in an exemplary embodiment of the present application;

fig. 2 is a schematic flowchart of a message processing method according to an exemplary embodiment of the present application;

fig. 3 is a schematic diagram of a message processing process according to an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of a process for determining a first attribute characteristic and a second attribute characteristic provided by an exemplary embodiment of the present application;

fig. 5 is a schematic structural diagram of a message processing apparatus according to an exemplary embodiment of the present application;

fig. 6 is a schematic structural diagram of a message processing device according to an exemplary embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in other sequences than those illustrated or described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Aiming at the problems that in the prior art, under the application scene of high concurrency and large flow, message replication needs great extra resources and affects the system performance, in the embodiment of the application, the attribute information statistics is carried out on the messages in the current statistical period through the sliding window, and for each attribute information, a first attribute characteristic corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the message corresponding to the attribute information in each sliding window in the current statistical period are determined, analyzing the message in the current statistical period according to the first attribute characteristic and the second attribute characteristic, the method can improve the precision of abnormal message detection, determine the scale of the abnormal message, does not need to relate to message copying among different devices, can save system resources, and performs message analysis in complex scenes of large flow, high concurrency, high-performance storage, mixed use and the like.

The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic structural diagram of a service system provided in an embodiment of the present application, and as shown in fig. 1, the service system includes: client 101, server 102, and storage node 103.

In this embodiment, the number of the clients 101, the servers 102 and the storage nodes 103 is not limited, and the number of the clients 101, the servers 102 and the storage nodes 103 may be the same or different. The number of the clients 101 may be one, or may be multiple, for example, 2, 5, or 20, etc.; similarly, the number of the server 102 may be one or more, the number of the storage nodes 103 may be one or more, and fig. 1 illustrates an example in which the numbers of the client 101, the server 102, and the storage nodes 103 are all plural, but the present invention is not limited thereto.

In this embodiment, the complex scenarios of large flow, high concurrency, and mixture of different high and low performance storage flows may include but are not limited to: a micro-service scenario, a big data scenario, a cloud scenario, etc., without limitation. In the following, the functions of each device in the service system will be described by taking a micro service as an example and a micro service scenario as an example, but the invention is not limited thereto. In a microservice scenario, one or more microservices are deployed on the server 102, each microservice is independent of another, and completes a service task by cooperation, and the client 101 can access data on the storage node 103 through the microservices on the server. The query of the same data sent by the client 101 to the microservice, that is, the access request for the same data, is finally routed to the same storage node 103, for the storage node 103, the QPS that the storage node 103 can carry by itself has an upper limit, and if an abnormal packet exists, the access request for the storage node 103 exceeds the upper limit, the stability of the storage node may be damaged, and even there is a risk of occurrence of avalanche.

At present, the abnormal message has great potential harm to the storage node, and the abnormal message and the abnormal access request need to be found when the abnormal message is treated. A method for discovering abnormal messages comprises the following steps: the flow of the main node is subjected to primary link replication, the message is sent to a special message acquisition and detection service, and the abnormal message is acquired on the detection service based on time slices, wherein the specific acquisition method comprises dynamic weighting full sequencing and the like, so that the abnormal message is screened out. Another method for discovering abnormal message volume is as follows: the messages are collected and recorded on the server side, and due to limited resources, a sorting elimination algorithm is required to sort the collected messages, and only the optimal solution of Top N before the access times is reserved. The currently Used sort-out algorithm is the Least recently Used algorithm (LFU) and Aging (Aging) frequency decay for recording.

The application also provides a message processing method, which is suitable for the server and used for analyzing the message accessing the server so as to find the abnormal message. As shown in fig. 2, the method at least comprises 201 and 203.

201. Receiving one or more first messages, wherein the first messages have one or more attribute information;

202. aiming at any kind of attribute information, determining a first attribute characteristic of the second message corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the second message corresponding to the attribute information in each sliding period corresponding to each sliding window; the second message is one or more first messages in the current statistical period;

203. and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

In this embodiment, the server may receive one or more first packets sent by different clients, where the number of each first packet may be 0 or one, or may be multiple, for example, 2, 3, or 5. The same type of first packet may access the same resource on the storage node through the server, where the same resource may be the same data Block (Block) or the same type of data on the storage node, which is not limited to this. Alternatively, as shown in fig. 3, a synchronous non-blocking queue may be set for each attribute information, or a synchronous non-blocking queue may be set for a plurality of kinds of attribute information. Fig. 3 illustrates an example in which there is one isochronous non-blocking queue corresponding to each attribute information, and 3 isochronous non-blocking queues are shown in total, but the present invention is not limited thereto.

One or more messages may be added to a target queue, also referred to as a synchronous non-blocking queue, in a synchronous non-blocking manner. Synchronous non-blocking synchronization means that after an I/O request for adding a message to a target queue is initiated, the I/O request is continuously initiated to add other messages to the target queue only after the I/O operation of a waiting or polling kernel is completed; synchronous non-blocking refers to that an I/O operation for adding a message to a target queue is called and then immediately returns a state value, the state value indicates that the action for adding the message to the target queue is executed, the I/O operation does not need to be completely completed, and the next message adding operation is executed after the message is added to the target queue. The method comprises the steps of adding messages to a synchronous non-blocking queue in a synchronous non-blocking mode, collecting messages of a main process, providing a basis for identification of subsequent abnormal messages, improving identification speed of the abnormal messages, and meanwhile, adding one or more messages to a target queue under the condition that the main process is not blocked, wherein the main process is mainly used for processing one or more messages, namely distributing the one or more messages to corresponding storage nodes. Under the condition that multiple kinds of attribute information correspond to one synchronous non-blocking queue, one or more kinds of first messages in a main process can be collected, the collected one or more kinds of first messages are added to the synchronous non-blocking queue in a synchronous non-blocking mode, and abnormal messages can be identified from multiple kinds of attribute information dimensions subsequently based on the first messages in the synchronous non-blocking queue. Under the condition that each attribute information corresponds to one synchronous non-blocking queue, for example, for three attribute information, three synchronous non-blocking queues may exist, one or more first messages in the main process may be collected, the collected first messages may be copied, the copied first messages are added to different synchronous non-blocking queues in a synchronous non-blocking manner, and subsequently, the identification of the abnormal message may be performed for the attribute information based on the first message in the synchronous non-blocking queue corresponding to each attribute information. Compared with the method that the first message is copied to other nodes except the computing node for identifying the abnormal message, the method reduces the process of message network transmission, saves network resources and improves the identification efficiency of the abnormal message.

In this embodiment, the one or more attribute information refers to attribute information of an abnormal packet, and the access request arrives at the server in the form of a packet. For the same access request, the message corresponding to the access request has the attribute information of the access request.

If the abnormal message is a message corresponding to the hotspot request, the abnormal message has attribute information of the hotspot request, for example, a characteristic of high frequency, and accordingly, the attribute characteristic of the abnormal message is the frequency of the abnormal message within a certain time. The hot spot request is an access request for frequently accessing the same resource on the storage node within a certain time, and an abnormal message corresponding to the hot spot request may cause that the access pressure of a single storage node is too large, and the pressure cannot be eliminated in a horizontal capacity expansion manner, and the storage node is unavailable in an extreme case.

If the abnormal message is a message corresponding to the large flow request, the abnormal message has attribute information of the large flow request, for example, a characteristic of a large data volume, and accordingly, the attribute characteristic of the abnormal message is the data volume of the abnormal message. The large-flow request refers to an access request that the data volume of a message exceeds a set data volume threshold within a certain time, the set data volume threshold may be 1GB, 5GB or 50GB, and the like, and specifically, depending on the access upper limit of the storage node, the large-flow request reaches the server in the form of a message and is finally routed to the storage node, and the large-flow request may cause the network outlet flow of the routed storage node to be consumed in a large amount and even exceed the upper limit of the outlet flow, thereby causing a serious negative effect on the performance of the storage node.

If the abnormal message is a message corresponding to the slow request, the abnormal message has attribute information of the slow request, for example, a high delay characteristic, and accordingly, the attribute characteristic of the abnormal message is the delay of the message. The slow request refers to an access request that is not responded beyond a set time threshold, and the set time threshold may be 30s, 50s, 70s, or the like. The slow request makes the access request not be responded in time, which affects the experience of the user.

In this embodiment, as shown in fig. 3, the identification of the abnormal packet may be performed in one or more statistical periods, and the duration of each statistical period is not limited, for example, the duration of each statistical period may be 3s, 5s, or 10 s. For convenience of description and distinction, one or more first messages in the current statistical cycle are denoted as second messages, and the number of each second message is not limited. In the process of analyzing or identifying the abnormal message, the sliding operation can be executed on the sliding window, and the attribute information is counted aiming at the second message in the current counting period. Optionally, the second packet in the current statistical period may be stored in a synchronous non-blocking queue, and the sliding window slides relative to the synchronous non-blocking queue, where the sliding manner is not limited, for example, the second packet in the synchronous non-blocking queue flows in and out from the sliding window in a First-in-First-out (FIFO) manner. The sliding window has attribute information, which includes but is not limited to: the sliding period is the sliding time of the sliding window, and the length corresponding to the sliding period is the sum of the sliding length of the sliding window in the current statistical period and the sliding length of the sliding window. As shown in fig. 4, a sliding process of a sliding window on a synchronous non-blocking queue is shown, where the length of the sliding window is 4, at a time t0 in a current statistical period, the sliding window is at a position X0, the sliding length is 0, at a time t0, the sliding period is 0, and the length corresponding to the sliding period is 4+0= 4; performing sliding operation on a sliding window, wherein the sliding step length of the sliding window is 1, the sliding window slides for two steps, the sliding window is located at a position X1, the current statistical period reaches the time t1, the sliding step length of the sliding window is 2, the sliding period is t1-t0 at this time, and the length corresponding to the sliding period is 4+2= 6; and performing sliding operation on the sliding window, wherein the sliding step of the sliding window is 1, the sliding window slides forwards by two steps, the sliding window reaches a position X2, when the current statistical period reaches t2, the sliding step of the sliding window is 4, at this time, the sliding period is t2-t0, and the length corresponding to the sliding period is 4+4= 8. The sliding period may be 10, 30 or 50, etc., and the sliding speed of the sliding window may be 100 ten thousand times per second or 1000 ten thousand times per second, etc.

In this embodiment, under the condition that each attribute information is allocated with one synchronous non-blocking queue, a sliding window may be allocated for each synchronous non-blocking queue, and a sliding operation is performed on the sliding window, so as to determine a first attribute feature and a second attribute feature of a second packet in a current statistical period; under the condition that multiple kinds of attribute information correspond to one synchronous non-blocking queue, a sliding window can be allocated to each kind of attribute information, the lengths, sliding speeds or sliding step lengths of different sliding windows can be different, and sliding operation is executed on the sliding windows according to the respective sliding step lengths, so that the first attribute characteristics and the second attribute characteristics of the second message aiming at each kind of attribute information in the current statistical period are counted.

In this embodiment, for any kind of attribute information, in each sliding window of the current statistical period, a first attribute feature of the second packet in each sliding window corresponding to the attribute information and a second attribute feature of the second packet in the sliding period corresponding to the attribute information in the current statistical period are determined, where the first attribute feature is an attribute feature of the second packet in each sliding window, and the second attribute feature is an attribute feature of the second packet in the sliding period in the current statistical period. It should be noted that, when determining the first attribute feature and the second attribute feature, the manner of determining the first attribute feature and the second attribute feature is different according to the difference of the attribute information. If the attribute information is the frequency of occurrence of the message, the first attribute information of the second message may be to count the frequency of occurrence of each second message in each sliding window for different types of second messages, and the second attribute feature of the second message is to count the frequency of occurrence of each second message in the sliding period for different types of second messages. If the attribute information is the time delay of the message, the first attribute information of the second message may be the time delay of each second message in each sliding window counted for different types of second messages, and the second attribute feature of the second message is the time delay of each second message in the sliding period counted for different types of second messages. If the attribute information is the data volume of the packet, the first attribute information of the second packet may be the data volume of each second packet in each sliding window counted for different types of second packets, and the second attribute feature of the second packet is the data volume of each second packet in the sliding period counted for different types of second packets.

As shown in fig. 4, the first attribute feature and the second attribute feature will be described by taking the frequency of the second packet as an example. When the sliding window is at the position X0, the second packet in the sliding window is [ a, B, a, B ], and the second packet in the sliding period is [ a, B, a, B ], and then the first attribute of the second packet is characterized by: the frequency of the A is 2, the frequency of the B is 2, and the second attribute characteristic of the second message is as follows: the frequency of A is 2, the frequency of B is 2; when the sliding window is at the position X1, the second packet in the sliding window is [ a, B, a, C ], and the second packet in the sliding period is [ a, B, a, C ], so that the first attribute characteristic of the second packet is: the frequency of A is 2, the frequency of B is 1, the frequency of C is 1, and the second attribute characteristic of the second message is as follows: the frequency of A is 3, the frequency of B is 2, and the frequency of C is 1; when the sliding window is at the position X2, the second packet in the sliding window is [ a, C, D ], and the second packet in the sliding period is [ a, B, a, C, D ], so that the first attribute of the second packet is: the frequency of A is 1, the frequency of C is 1, the frequency of D is 2, and the second attribute characteristic of the second message is as follows: the frequency of A is 3, the frequency of B is 2, the frequency of C is 1, and the frequency of D is 2. In this embodiment, with the continuous sliding of the sliding window, the first attribute feature and the second attribute feature of the second packet in the current statistical period are continuously changed, and the abnormal degree of the second packet in the current statistical period may be determined according to the first attribute feature and the second attribute feature of the second packet. For example, in the current statistical period, dynamic parameters of different types of second messages are determined according to the first attribute features and the second attribute features, comprehensive attribute features of the different types of second messages are determined based on the dynamic parameters, and if the comprehensive attribute features of the second messages meet set conditions, the second messages are considered to be abnormal messages, for example, the dynamic parameters may be implemented as sorting weights referred to in the following description, which is specifically referred to in the following embodiments and will not be described in detail here. As shown in fig. 3, for each attribute information, the identification result of the abnormal degree of the second packet in the current statistical period may be stored in a result queue, and a plurality of result queues are output to a storage device, for example, the storage device may include, but is not limited to, a hard disk or a database.

In the embodiment of the application, the attribute information statistics is performed on the messages in the current statistical period through the sliding window, for each attribute information, the first attribute characteristic of the messages in the sliding period corresponding to each sliding window in the current statistical period and the second attribute characteristic of the messages in each sliding window in the current statistical period are determined, the messages in the current statistical period are analyzed according to the first attribute characteristic and the second attribute characteristic, the sliding operation is performed on the sliding window compared with a common LFU algorithm or sorting elimination algorithm such as Aging frequency attenuation, and the like, the first attribute characteristic and the second attribute characteristic are obtained through statistics, so that the abnormal degree of the second messages is determined, the detection precision of the abnormal messages can be improved, the scale of the abnormal messages is determined, in addition, the second messages in the current statistical period are added into the synchronous non-blocking queue without copying and network transmission, the method saves network resources and storage resources of the system, can perform message analysis in complex scenes of large flow, high concurrency, different high and low performance flow storage mixed use and the like, and solves the problem that the traditional abnormal message detection algorithm cannot be applied to the increasingly complex scene abnormal message collection of large flow, high concurrency and diversified rear-end storage engines due to single scene.

In an optional embodiment, a sliding operation may be performed on the sliding window, and for any attribute information, statistics of the attribute information is performed on the second packet in the current statistics period, so as to obtain first attribute features of the second packet corresponding to the attribute information in each sliding window and second attribute information corresponding to the attribute information in the sliding period corresponding to each sliding window in the current statistics period. In this way, each sliding operation corresponds to one sliding window, and a plurality of sliding windows may be included in the current statistical period. For convenience of description and distinction, a sliding window corresponding to the current sliding operation is referred to as a current sliding window, a sliding operation before the current sliding operation in the current statistical period is referred to as a historical sliding operation, a sliding window corresponding to the historical sliding operation is referred to as a historical sliding window, a second message in the current sliding window is referred to as a third message, a second message corresponding to the historical sliding window and the current sliding window is referred to as a fourth message, a message in the sliding period can be determined according to the historical sliding window and the second message in the current sliding window, and the second message in the sliding period is referred to as the fourth message.

The embodiment of determining the fourth packet in the sliding cycle according to the second packet in the history sliding window and the current sliding window is not limited, but the determination of the fourth packet is different according to the difference between the sliding step of the sliding operation and the length of the sliding window, which will be described in detail below.

And under the condition that the sliding step length of the sliding operation is equal to the length of the sliding window, the sliding operation is executed aiming at the sliding window, the same message does not repeatedly appear in the current sliding window and the historical sliding window, and the message in the sliding period can be the sum of the message in each historical sliding window and the message in the current sliding window. It should be noted that the same message may be repeated in the current sliding window and the historical sliding window, but the same message may not be repeated, for example, in fig. 4, if the sliding step is equal to the length of the sliding window, the synchronous non-blocking queue in fig. 4 may include two sliding windows, one is the sliding window at the position X0, and the other is the sliding window at the position X2, the same message a may appear twice in the sliding window at the position X0, but two messages a in the sliding window at the position X0 may not appear in the sliding window at the position X2, if the sliding window at the position X0 is not the historical sliding window, and the sliding window at the position X2 is the current sliding window, then the same message may not repeatedly appear in the current sliding window and the historical sliding window.

In the case that the sliding step of the sliding operation is smaller than the length of the sliding window, the sliding operation is performed on the sliding window, and there is an overlap between the messages in the current sliding window and the messages in the historical sliding window, as shown in fig. 4, the sliding step of the sliding window is 2, which is smaller than the length 4 of the sliding window, and the third message a and the fourth message B located in the sliding window from left to right at the time t0 repeatedly appear in the sliding window at the time t1, specifically, the first message a and the second message B in the sliding window at the time t1, so the messages in the sliding cycle may be the messages repeated between two sliding windows subtracted from the sum of the messages in each historical sliding window and the messages in the current sliding window.

When the sliding step length of the sliding operation is greater than the length of the sliding window, the sliding operation is executed for the sliding window, the same message in the current sliding window and the historical sliding window does not repeat, and other messages also exist between the current sliding window and the historical sliding window, so that the message in the sliding period can be the sum of the message in each historical sliding window and the message in the current sliding window plus the other messages between two adjacent sliding windows.

In this embodiment, for the first sliding operation, a first attribute feature of a third packet in a current sliding window and a second attribute feature of a fourth packet in a sliding period may be determined, and since the first sliding operation does not exist in a historical sliding window, a suspected abnormal packet in the third packet may be identified according to the first attribute feature of the third packet if the third packet in the current sliding window is the same as the fourth packet in the sliding period, and the suspected abnormal packet is used as the suspected abnormal packet identified by the current sliding operation. Under the condition that the sliding operation is not the first sliding operation, aiming at the sliding operation, the first attribute characteristic of the third message and the second attribute characteristic of the fourth message can be determined, and the suspected abnormal message identified by the sliding operation at the last time and the third message are identified according to the first attribute characteristic of the third message and the second attribute characteristic of the fourth message, so that the suspected abnormal message identified by the sliding operation at the time is obtained; and obtaining the abnormal message aiming at the attribute information in the current statistical period according to the suspected abnormal message identified by the last sliding operation in the current statistical period.

In an optional embodiment, the second attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period may be initialized by using the second attribute feature of the fourth packet corresponding to the last sliding operation in the previous statistical period, so as to obtain an initial attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period, and the second attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period is determined based on the initial attribute feature.

Specifically, according to the second attribute feature of the fourth packet corresponding to the last sliding operation when the previous statistical period is ended, the initial attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period is determined. For example, the second attribute feature of the fourth packet corresponding to the last sliding operation at the end of the previous statistical period may be used as the initial attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period, or the second attribute feature of the fourth packet corresponding to the last sliding operation at the end of the previous statistical period may be multiplied by the set smoothing coefficient to obtain the initial attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period. The smoothing coefficient represents the degree of influence of the second attribute characteristic of the fourth packet corresponding to the last sliding operation in the previous statistical period on the second attribute characteristic of the fourth packet corresponding to the current sliding operation in the current statistical period, and the smoothing coefficient may take a value between [0 and 1 ].

After the initial attribute characteristic of the fourth message corresponding to the sliding operation of this time in the current statistical period is determined, the second attribute characteristic of the fourth message corresponding to the sliding operation of this time in the current statistical period is determined based on the initial attribute characteristic. For example, the second attribute feature of the fourth packet corresponding to the current sliding operation in the current statistical period may be calculated through a function P0= α. P0 represents a second attribute feature of a fourth message corresponding to the current sliding operation in the current statistical period, V _ old represents a second attribute feature of a fourth message corresponding to the last sliding operation in the previous statistical period, and V _ new represents initial second attribute information of the fourth message corresponding to the current sliding operation in the current statistical period; "." denotes multiplication; and alpha is a smoothing coefficient, a value between [0 and 1] is taken, the value is determined according to an application scene, and when alpha =0, the second attribute information of the fourth message corresponding to the sliding operation of this time in the current statistical period is represented, is irrelevant to the second attribute characteristic of the fourth message corresponding to the last sliding operation in the previous statistical period, and is relevant to the initial second attribute information of the fourth message corresponding to the sliding operation of this time in the current statistical period. Where α, V _ old represents the initialized property feature.

In an optional embodiment, an implementation manner of identifying, according to a first attribute feature and a second attribute feature, a third packet and a suspected abnormal packet identified by a last sliding operation to obtain the suspected abnormal packet identified by the current sliding operation includes: and in the current statistical period, aiming at each sliding operation, determining the suspected abnormal message identified by the sliding operation from the third message and the abnormal message identified by the sliding operation at the last time according to the second attribute characteristics of the suspected abnormal message identified by the third message and the sliding operation at the last time and the length of the sliding window. For example, when the second attribute feature is a frequency, the length of the sliding window, the frequency of the third packet, and the frequency of the suspected abnormal packet identified by the previous sliding operation are determined, the frequencies of the third packet and the suspected abnormal packet are compared with the length of the sliding window, and if the frequency is greater than or equal to half the length of the sliding window, the packet corresponding to the frequency information is considered as the suspected abnormal packet identified by the current sliding operation. As shown in fig. 4, the second attribute is frequency, which is represented by count, the length of the sliding window is 4, and at the current position X2, the frequencies of the third message A, C and D in the sliding window, and the frequencies of A, C and D are 3,1, and 2, respectively, where the suspected message identified by the previous sliding operation is: a and B, the frequencies of A and B being 3 and 2. At the current position X2, a in the third message is the suspected message identified by the previous sliding operation, the frequency 1 of C is less than half (1/2) × 4=2 of the sliding window, and the frequency 2 of D is equal to half (1/2) × 4=2 of the sliding window, so the suspected abnormal messages identified by the current sliding operation are the suspected abnormal messages identified by the previous sliding operation and the suspected abnormal messages in the third message, namely A, B and D. For another example, when the second attribute feature is a time delay, the length of the sliding window, the time delay of the third packet, and the time delay of the suspected abnormal packet identified by the last sliding operation are determined, the time delays of the same packet in the third packet and the suspected abnormal packet are compared with the length of the sliding window, and if the time delay is greater than or equal to the length of the sliding window, the packet corresponding to the time delay is considered as the suspected abnormal packet identified by the current sliding operation. For example, the length of the sliding window is 4, and the suspected abnormal message identified by the last sliding operation is: e and F, the time delays of E and F are 4 and 3, respectively, the third messages in the current sliding window are K, L and M, and the time delays of K, L and M are: 1. 2 and 3, half (1/2) × 4=2 of the sliding window, and the

time delays

2 and 3 of L and M are both greater than half of the sliding window, so that the suspected abnormal messages in the third message are L and M, and the suspected abnormal messages identified by the sliding operation are E, F, L and M.

In an optional embodiment, when a suspected exception packet is identified in the current statistical period, the suspected exception packet is added to a buffer, which may also be referred to as a protection zone (protection). The protection area is provided with a message adding strategy and a message eliminating strategy.

The adding strategy is a strategy of adding the third message to the cache region as a suspected abnormal message when the second attribute characteristic of the third message in the sliding window and the length of the sliding window meet set conditions. For example, the addition policy may be: if the second characteristic attribute of the third packet is greater than half of the length of the sliding window, an addition policy is triggered, and the packet may be attempted to be added to the protection area. Specifically, the method comprises the following steps: after the message enters the sliding window, executing an adding strategy to the message: and adding the message with the second attribute characteristic being larger than half of the length of the sliding window into the protection area, and not adding the message with the second attribute characteristic being not larger than half of the length of the sliding window into the protection area. For example: for the message 1 and the message 2, when the second characteristic attribute of the message 1 entering the sliding window is larger than half of the length of the sliding window, the message 1 is added to the protection area as a suspected abnormal message, and when the second characteristic attribute of the message 2 entering the sliding window is not larger than half of the length of the sliding window, the message 2 is not added to the protection area.

The elimination strategy refers to a strategy of deleting suspected abnormal messages from the cache area according to dynamic parameters (such as sorting weight) of the suspected abnormal messages in the protection area. And when the suspected abnormal message of the sliding operation is identified, adding the suspected abnormal message to the cache region, and when the message leaves the sliding window, executing a elimination strategy and deleting the message in the cache region. Specifically, after the message leaves the sliding window, a knockout strategy is executed on the message, and the sorting weight of the suspected abnormal message is determined according to the first attribute feature and the second attribute feature of the suspected abnormal message in the cache region, wherein the sorting weight of the suspected abnormal message can be obtained by performing weighted summation on the first attribute feature and the second attribute feature of the suspected abnormal message, or can be obtained by performing operation on the first attribute feature and the second attribute feature of the suspected abnormal message in other mathematical modes, and is not limited; after the sorting weight of the suspected abnormal messages is determined, when the number of the suspected abnormal messages in the cache area exceeds a first set number, the suspected abnormal messages are filtered according to the sorting weight of the suspected abnormal messages in the cache area, so that the first set number of the suspected abnormal messages are reserved. The first set number may be a number smaller than or equal to the length of the buffer, for example, the first set number may be 2, 3, or 5. For example, after a message leaves the sliding window, the suspected abnormal messages in the cache area may be sorted according to the sorting weight of the suspected abnormal messages, where the suspected abnormal messages and the sorting weight thereof are: t1:5, T2:4, T3:2, the length of the cache region is 3, the number of suspected abnormal messages needing to be reserved in the cache region is 2, the suspected abnormal messages T3 are deleted from the cache region according to the sorting weight, and suspected abnormal messages T1 and T2 are reserved in the cache region, so that when suspected abnormal messages are added to the cache region, the suspected abnormal messages in the cache region cannot overflow.

In an alternative embodiment, as shown in fig. 4, in addition to the buffer (protection) area, an attribute feature statistical mapping table (freqMap) is provided. The data recorded in the statistical mapping table are the first attribute feature and the second attribute feature of the third message of the sliding window and the first attribute feature and the second attribute feature of the suspected abnormal message in the cache region. The first attribute characteristic and the second attribute characteristic of the suspected abnormal message in the statistical mapping table are continuously changed, the first attribute characteristic and the second attribute characteristic of the suspected abnormal message can be timely and quickly updated through the statistical mapping table, the suspected abnormal message is timely synchronized into the cache region through the adding strategy and the eliminating strategy of the cache region, and the abnormal message analysis efficiency is improved.

The method and the device introduce a new dynamic weight sorting algorithm innovatively, namely a Full window Least-recently-Frequently-accessed (FW-LFU) algorithm, perform sliding operation on a sliding window, dynamically determine a first attribute characteristic and a second attribute characteristic of the message, and count the second message in the current counting period according to the first attribute characteristic and the second attribute characteristic of the suspected abnormal message, so as to determine the suspected abnormal message in the current counting period. Adding the suspected abnormal messages into the cache region through an adding strategy, sorting the suspected abnormal messages in the cache region according to the sorting weight of the suspected abnormal messages in the cache region through an eliminating strategy, deleting the suspected abnormal messages of the set number after sorting, thereby reserving the suspected abnormal messages of the first set number, and taking the suspected abnormal messages of the first set number as the suspected abnormal messages in the current statistical period. The time complexity of the FW-LFU algorithm is O (N × WlogW), the space complexity is O (X +2W), N is the number of second messages in the current statistical period, X is the length of the cache region, W is the length of the sliding window, and the order of magnitude of the space complexity and the time complexity is represented by O. Compared with the traditional LFU algorithm, the space complexity and the time complexity of the FW-LFU algorithm are improved to a certain extent, the efficiency is improved, the time is saved, and meanwhile, the storage space is saved. As shown in fig. 4, the number of second messages in the current statistical period is 9, the length of the sliding window is 4, the second messages of different types are represented by "A, B, C and D", and in fig. 4, it is described by taking an example that an abnormal message is a hot traffic and attribute information is the frequency of the message, it should be understood that a first attribute feature and a second attribute feature of the message are also the frequency of the message, where the second attribute feature is represented by a total number of times (count), and the first attribute feature is represented by a window Weight (Windows Weight, ww). The sorting weight of the suspected abnormal messages in the cache region is the sum of the total times and the window weight, namely count + ww. As shown in fig. 4, the length of the cache region is 2, when the cache region is full, that is, when the number of suspected abnormal messages in the cache region exceeds 2, according to the sorting weight of the suspected abnormal messages, the suspected abnormal messages with the lowest sorting weight are preferentially removed from the cache region, the suspected abnormal messages stored in the current cache region are a and B, the sorting weights of a and B are 4 and 2, respectively, and the sorting weight of an access request D that needs to be newly added to the cache region is 4, then the access request B is preferentially removed from the cache region. And counting the total times of the third message of the current sliding window and the suspected abnormal messages in the cache region and the window weight stored in the mapping table.

In this embodiment, according to the first attribute feature of the third packet and the second attribute feature of the fourth packet in the current statistical period, the abnormal packet for the attribute information in the current statistical period is obtained and output to the outside. The abnormal messages corresponding to the attribute information can be output to the outside in each statistical period, and the abnormal messages identified in the statistical periods can be aggregated to obtain the abnormal messages corresponding to the statistical periods. The number of the plurality of statistical cycles is not limited, for example, the number of the plurality of statistical cycles may be 10, 20, or 30, etc. Based on the above, in some optional embodiments of the present application, for a plurality of statistical cycles corresponding to any attribute information, a sorting weight of an abnormal packet for a certain attribute information in each statistical cycle is determined, where the sorting weight of the abnormal packet may be determined by a first attribute feature and a second attribute feature of the abnormal packet corresponding to a last sliding operation in a current statistical cycle, for example, the sorting weight of the abnormal packet may be obtained by weighted summation of the first attribute feature and the second attribute feature. After determining the sorting weight of the abnormal messages for a certain attribute information, selecting a second set number of fifth messages from the abnormal messages for the attribute information in the statistical period according to the sorting weight, where the fifth messages include at least one abnormal message, and for example, the fifth messages with a second set number with a higher sorting weight may be selected from the abnormal messages for the attribute information in the statistical period from high to low, and the second set number may be a number less than or equal to the length of the cache region, for example, the second set number may be 3, 5, or 7, and the like, which is not limited thereto; and performing attribute aggregation on the same abnormal message in the fifth message selected from the multiple statistical periods to obtain aggregation information of each abnormal message in the multiple statistical periods, and outputting the aggregation information of each abnormal message aiming at the attribute information. For example, the aggregation information of each packet may be stored in a result queue, so as to obtain a plurality of result queues, and the plurality of result queues are output to a storage device, for example, the storage device may include, but is not limited to, a hard disk or a database.

Optionally, the embodiment of performing attribute aggregation on the same abnormal packet in the fifth packet selected from the multiple statistical periods to obtain aggregation information of each abnormal packet in the multiple statistical periods includes: aiming at any abnormal message, determining the aggregation information of the abnormal message in the current statistical period according to the second attribute information, the smoothing coefficient and the aggregation information of the abnormal message in the previous statistical period; and obtaining the aggregation information of the abnormal message in a plurality of continuous statistical periods according to the aggregation information of the abnormal message in the last statistical period in the plurality of continuous statistical periods.

For example, it can be implemented by a function P_i=α•P_i-1 +(1-α)• P’_iPerforming attribute aggregation on the same abnormal message in the fifth message, wherein i is more than or equal to 2, and P_iRepresenting the aggregation information of the abnormal message in the current statistical period; p_i-1Representing the aggregation information of the abnormal message in the previous statistical period; p'_iRepresenting the initial aggregation information of the abnormal message in the current statistical period; alpha is a smoothing coefficient, and is [0,1]]The value between the first and second values is specifically determined according to an application scenario, and when α =0, it indicates that the aggregation information of the abnormal packet in the fifth packet in the current statistical period is related to the initial aggregation information of the abnormal packet in the current statistical period, and is unrelated to the aggregation information of the abnormal packet in the previous statistical period.

It should be noted that the execution subjects of the steps of the methods provided in the above embodiments may be the same device, or different devices may be used as the execution subjects of the methods. For example, the execution subjects of step 201 to step 203 may be device a; for another example, the execution subject of steps 201 and 202 may be device a, and the execution subject of step 203 may be device B; and so on.

In addition, in some of the flows described in the above embodiments and the drawings, a plurality of operations are included in a specific order, but it should be clearly understood that the operations may be executed out of the order presented herein or in parallel, and the sequence numbers of the operations, such as 201, 202, etc., are merely used for distinguishing different operations, and the sequence numbers do not represent any execution order per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit the types of "first" and "second".

Fig. 5 is a schematic structural diagram of a message processing apparatus according to an exemplary embodiment of the present application. As shown in fig. 5, the message processing apparatus includes: a receiving module 51, a first determining module 52 and a second determining module 53.

A receiving module 51, configured to receive one or more first messages, where the first messages have one or more types of attribute information;

a first determining module 52, configured to determine, for any kind of attribute information, a first attribute feature, corresponding to the attribute information, of the second packet in each sliding window in the current statistical period, and a second attribute feature, corresponding to the attribute information, of the second packet in each sliding period corresponding to each sliding window; the second message is one or more first messages in the current statistical period;

and the second determining module 53 is configured to determine an abnormal degree of the second packet in the current statistical period according to the first attribute feature and the second attribute feature.

In an optional embodiment, the first determining module 52 is specifically configured to: executing sliding operation on the sliding window, counting attribute information of a second message in a current counting period, and determining a first attribute characteristic of a third message in the current sliding window and a second attribute characteristic of a fourth message in a historical sliding window and the current sliding window according to the current sliding operation; correspondingly, the second determining module 53 is specifically configured to: according to the first attribute characteristic and the second attribute characteristic, identifying the third message and the suspected abnormal message identified by the last sliding operation to obtain the suspected abnormal message identified by the sliding operation; and obtaining the abnormal message aiming at the attribute information in the current statistical period according to the suspected abnormal message identified by the last sliding operation in the current statistical period.

In an optional embodiment, the first determining module 52 is further configured to: and if the sliding operation is the first sliding operation, identifying the third message according to the first attribute characteristic to obtain a suspected abnormal message identified by the sliding operation.

In an optional embodiment, the first determining module 52 is specifically configured to: determining the initial attribute characteristic of the fourth message corresponding to the sliding operation in the current statistical period according to the second attribute characteristic of the fourth message corresponding to the last sliding operation when the previous statistical period is finished; and determining a second attribute characteristic of a fourth message corresponding to the sliding operation in the current statistical period based on the initial attribute characteristic.

In an optional embodiment, the second determining module 53 is specifically configured to: and in the current statistical period, aiming at each sliding operation, determining the suspected abnormal message identified by the sliding operation from the third message and the suspected abnormal message identified by the last sliding operation according to the second attribute characteristics of the suspected abnormal message identified by the third message and the last sliding operation and the length of the sliding window.

In an optional embodiment, the message processing apparatus further includes: the adding module, the third determining module and the filtering module; the adding module is used for adding the suspected abnormal message to the cache region when the suspected abnormal message of the sliding operation is identified; the third determining module is used for determining the sorting weight of the suspected abnormal message according to the first attribute characteristic and the second attribute characteristic of the suspected abnormal message; and the filtering module is used for filtering the suspected abnormal messages according to the sorting weight of the suspected abnormal messages in the cache region when the number of the suspected abnormal messages in the cache region exceeds a first set number, and reserving the first set number of the suspected abnormal messages.

In an optional embodiment, the message processing apparatus further includes: a selection module and an aggregation module; the selection module is used for selecting a second set number of fifth messages from the abnormal messages aiming at the attribute information in each statistical period according to the sorting weight of the abnormal messages aiming at the attribute information in each statistical period, wherein the fifth messages comprise at least one abnormal message; and the aggregation module is used for performing attribute aggregation on the same abnormal message in the fifth message selected from the plurality of statistical cycles to obtain aggregation information of each abnormal message in the plurality of statistical cycles.

In an alternative embodiment, the aggregation module is specifically configured to: aiming at any abnormal message, determining the aggregation information of the abnormal message in the current statistical period according to the second attribute information, the smoothing coefficient and the aggregation information of the abnormal message in the previous statistical period; and obtaining the aggregation information of the abnormal message in a plurality of continuous statistical periods according to the aggregation information of the abnormal message in the last statistical period in the plurality of continuous statistical periods.

The message processing device provided in the embodiment of the application performs attribute information statistics on the messages in the current statistical period through the sliding window, determines, for each attribute information, a first attribute feature corresponding to the attribute information in each sliding window in the current statistical period and a second attribute feature of the message corresponding to the attribute information in each sliding window in the current statistical period, and analyzes the messages in the current statistical period according to the first attribute feature and the second attribute feature, so that the accuracy of abnormal message detection can be improved, the scale of the abnormal messages can be determined, message copying among different devices is not required to be designed, system resources can be saved, and message analysis can be performed in complex scenes such as large flow, high concurrency, high-performance storage and mixing.

Fig. 6 is a schematic structural diagram of a message processing device according to an exemplary embodiment of the present application. As shown in fig. 6, the apparatus includes: a memory 64 and a processor 65.

The memory 64 is used to store computer programs and may be configured to store various other data to support operations on the message processing device. Examples of such data include instructions for any application or method operating on the message processing device.

The memory 64 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 65, coupled to the memory 64, for executing computer programs in the memory 64 for: receiving one or more first messages, wherein the first messages have one or more attribute information; aiming at any kind of attribute information, determining a first attribute characteristic of the second message corresponding to the attribute information in each sliding window in the current statistical period and a second attribute characteristic of the second message corresponding to the attribute information in each sliding period corresponding to each sliding window; the second message is one or more first messages in the current statistical period; and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

In an optional embodiment, the processor 65 determines a first attribute feature corresponding to the attribute information in each sliding window of the second packet in the current statistical period and a second attribute feature corresponding to the attribute information in each sliding period of the second packet in each sliding window; when the second packet is one or more first packets in the current statistical period, the method is specifically configured to: executing sliding operation on the sliding window, counting attribute information of a second message in a current counting period, and determining a first attribute characteristic of a third message in the current sliding window and a second attribute characteristic of a fourth message in a historical sliding window and the current sliding window according to the current sliding operation; correspondingly, when determining the abnormal degree of the second packet in the current statistical period according to the first attribute feature and the second attribute feature, the processor 65 is specifically configured to: according to the first attribute characteristic and the second attribute characteristic, identifying the third message and the suspected abnormal message identified by the last sliding operation to obtain the suspected abnormal message identified by the sliding operation; and obtaining the abnormal message aiming at the attribute information in the current statistical period according to the suspected abnormal message identified by the last sliding operation in the current statistical period.

In an alternative embodiment, the processor 65 is further configured to: and if the sliding operation is the first sliding operation, identifying the third message according to the first attribute characteristic to obtain a suspected abnormal message identified by the sliding operation.

In an optional embodiment, when determining the second attribute characteristic of the fourth packet located in the history sliding window and the current sliding window, the processor 65 is specifically configured to: determining the initial attribute characteristic of the fourth message corresponding to the sliding operation in the current statistical period according to the second attribute characteristic of the fourth message corresponding to the last sliding operation when the previous statistical period is finished; and determining a second attribute characteristic of a fourth message corresponding to the sliding operation in the current statistical period based on the initial attribute characteristic.

In an optional embodiment, when the processor 65 identifies the third packet and the suspected abnormal packet identified by the last sliding operation according to the first attribute feature and the second attribute feature, and obtains the suspected abnormal packet identified by the current sliding operation, the processor is specifically configured to: and in the current statistical period, aiming at each sliding operation, determining the suspected abnormal message identified by the sliding operation from the third message and the suspected abnormal message identified by the last sliding operation according to the second attribute characteristics of the suspected abnormal message identified by the third message and the last sliding operation and the length of the sliding window.

In an alternative embodiment, the processor 65 is further configured to: when a suspected abnormal message of the sliding operation is identified, adding the suspected abnormal message to a cache region; determining the sorting weight of the suspected abnormal message according to the first attribute characteristic and the second attribute characteristic of the suspected abnormal message; and when the number of the suspected abnormal messages in the cache region exceeds a first set number, filtering the suspected abnormal messages according to the sorting weight of the suspected abnormal messages in the cache region, and reserving the suspected abnormal messages of the first set number.

In an alternative embodiment, the processor 65 is further configured to: selecting a second set number of fifth messages from the abnormal messages aiming at the attribute information in each statistical period according to the sorting weight of the abnormal messages aiming at the attribute information in each statistical period, wherein the fifth messages comprise at least one abnormal message; and performing attribute aggregation on the same abnormal message in the fifth message selected from the plurality of statistical periods to obtain the aggregation information of each abnormal message in the plurality of statistical periods.

In an optional embodiment, when performing attribute aggregation on the same abnormal packet in the fifth packet selected from the multiple statistical periods to obtain aggregation information of each abnormal packet in the multiple statistical periods, the processor 65 is specifically configured to: aiming at any abnormal message, determining the aggregation information of the abnormal message in the current statistical period according to the second attribute information, the smoothing coefficient and the aggregation information of the abnormal message in the previous statistical period; and obtaining the aggregation information of the abnormal message in a plurality of continuous statistical periods according to the aggregation information of the abnormal message in the last statistical period in the plurality of continuous statistical periods.

According to the message processing device provided by the embodiment of the application, the attribute information statistics is performed on the messages in the current statistical period through the sliding windows, the first attribute characteristics corresponding to the attribute information in each sliding window in the current statistical period and the second attribute characteristics of the messages corresponding to the attribute information in the sliding period corresponding to each sliding window in the current statistical period are determined for each attribute information, and the messages in the current statistical period are analyzed according to the first attribute characteristics and the second attribute characteristics, so that the accuracy of abnormal message detection can be improved, the scale of the abnormal messages is determined, message copying among different devices is not required to be designed, system resources can be saved, and message analysis can be performed in complex scenes such as large flow, high concurrency, high-performance storage and mixing.

Further, as shown in fig. 6, the message processing apparatus further includes: communication components 66, display 67, power components 68, audio components 69, and the like. Only some of the components are schematically shown in fig. 6, and it is not meant that the message processing apparatus includes only the components shown in fig. 6. It should be noted that the components within the dashed box in fig. 6 are optional components, not necessary components, and may be determined according to the product form of the message processing apparatus.

Accordingly, embodiments of the present application also provide a computer readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of the method shown in fig. 2.

Accordingly, embodiments of the present application also provide a computer program product, which includes a computer program/instructions, and when executed by a processor, causes the processor to implement the steps of the method shown in fig. 2.

The communication component of fig. 6 described above is configured to facilitate communication between the device in which the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as a WiFi, a 2G, 3G, 4G/LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

The display in fig. 6 described above includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

The power supply assembly of fig. 6 described above provides power to the various components of the device in which the power supply assembly is located. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power component is located.

The audio component of fig. 6 described above may be configured to output and/or input an audio signal. For example, the audio component includes a Microphone (MIC) configured to receive an external audio signal when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in a memory or transmitted via a communication component. In some embodiments, the audio assembly further comprises a speaker for outputting audio signals.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A message processing method is characterized by comprising the following steps:

receiving one or more first messages, wherein the first messages have one or more attribute information;

performing sliding operation on the sliding window according to any attribute information, and performing attribute information statistics on a second message in a current statistics period to determine a first attribute feature of the second message corresponding to the attribute information in each sliding window in the current statistics period and a second attribute feature of the second message corresponding to the attribute information in each sliding period corresponding to the sliding window; the second message is one or more first messages in the current statistical period;

and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

2. The method of claim 1, wherein performing a sliding operation on sliding windows, and performing attribute information statistics on a second packet in a current statistics period to determine a first attribute characteristic corresponding to the attribute information in each sliding window of the second packet in the current statistics period and a second attribute characteristic corresponding to the attribute information in each sliding period of the second packet in each sliding window of the second packet in the current statistics period comprises:

executing sliding operation on the sliding window, counting attribute information of a second message in a current counting period, and determining a first attribute characteristic of a third message in the current sliding window and a second attribute characteristic of a fourth message in a historical sliding window and the current sliding window according to the current sliding operation;

correspondingly, determining the abnormal degree of the second message in the current statistical period according to the first attribute feature and the second attribute feature, including:

according to the first attribute feature and the second attribute feature, identifying the third message and the suspected abnormal message identified by the last sliding operation to obtain the suspected abnormal message identified by the sliding operation;

and obtaining the abnormal message aiming at the attribute information in the current statistical period according to the suspected abnormal message identified by the last sliding operation in the current statistical period.

3. The method of claim 2, further comprising: and if the sliding operation is the first sliding operation, identifying the third message according to the first attribute characteristic to obtain a suspected abnormal message identified by the sliding operation.

4. The method of claim 2, wherein determining the second attribute characteristic of the fourth packet in the history sliding window and the current sliding window comprises:

determining the initial attribute characteristic of the fourth message corresponding to the sliding operation in the current statistical period according to the second attribute characteristic of the fourth message corresponding to the last sliding operation when the previous statistical period is finished;

and determining a second attribute characteristic of a fourth message corresponding to the sliding operation in the current statistical period based on the initial attribute characteristic.

5. The method according to claim 4, wherein identifying the third packet and the suspected abnormal packet identified by the last sliding operation according to the first attribute feature and the second attribute feature to obtain the suspected abnormal packet identified by the current sliding operation includes:

and in the current statistical period, aiming at each sliding operation, determining the suspected abnormal message identified by the sliding operation at this time from the third message and the suspected abnormal message identified by the last sliding operation according to the second attribute characteristics of the suspected abnormal message identified by the third message and the last sliding operation and the length of the sliding window.

6. The method of claim 5, further comprising:

when a suspected abnormal message of the sliding operation is identified, adding the suspected abnormal message to a cache region;

determining the sorting weight of the suspected abnormal message according to the first attribute characteristic and the second attribute characteristic of the suspected abnormal message;

and when the number of suspected abnormal messages in the cache region exceeds a first set number, filtering the suspected abnormal messages according to the sorting weight of the suspected abnormal messages in the cache region, and reserving the suspected abnormal messages of the first set number.

7. The method of claim 6, further comprising:

selecting a second set number of fifth messages from the abnormal messages aiming at the attribute information in each statistical period according to the sorting weight of the abnormal messages aiming at the attribute information in each statistical period, wherein the fifth messages comprise at least one abnormal message;

and performing attribute aggregation on the same abnormal message in the fifth message selected from the plurality of statistical periods to obtain aggregation information of each abnormal message in the plurality of statistical periods.

8. The method according to claim 7, wherein performing attribute aggregation on the same abnormal packet in a fifth packet selected from the plurality of statistical periods to obtain aggregation information of each abnormal packet in the plurality of statistical periods comprises:

aiming at any abnormal message, determining the aggregation information of the abnormal message in the current statistical period according to the second attribute information, the smoothing coefficient and the aggregation information of the abnormal message in the previous statistical period; and obtaining the aggregation information of the abnormal message in a plurality of continuous statistical periods according to the aggregation information of the abnormal message in the last statistical period in the plurality of continuous statistical periods.

9. A message processing apparatus, comprising: the device comprises a receiving module, a first determining module and a second determining module;

the receiving module is used for receiving one or more first messages, and the first messages have one or more attribute information;

the first determining module is configured to perform a sliding operation on the sliding window according to any attribute information, perform attribute information statistics on a second packet in a current statistics period, and determine a first attribute feature of the second packet corresponding to the attribute information in each sliding window in the current statistics period and a second attribute feature of the second packet corresponding to the attribute information in each sliding period corresponding to the sliding window; the second message is one or more first messages in the current statistical period;

and the second determining module is used for determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

10. The apparatus of claim 9, wherein the first determining module is specifically configured to:

correspondingly, the second determining module is specifically configured to:

11. The apparatus of claim 10, wherein the second determining module is further configured to: and if the sliding operation is the first sliding operation, identifying the third message according to the first attribute characteristic to obtain a suspected abnormal message identified by the sliding operation.

12. The apparatus of claim 10, wherein the first determining module is specifically configured to:

determining the initial attribute characteristic of the fourth message corresponding to the sliding operation at this time in the current statistical cycle according to the second attribute characteristic of the fourth message corresponding to the last sliding operation when the previous statistical cycle is finished;

13. The apparatus of claim 12, wherein the second determining module is specifically configured to: and in the current statistical period, aiming at each sliding operation, determining the suspected abnormal message identified by the sliding operation at this time from the third message and the suspected abnormal message identified by the last sliding operation according to the second attribute characteristics of the suspected abnormal message identified by the third message and the last sliding operation and the length of the sliding window.

14. The apparatus of claim 13, further comprising: the adding module, the third determining module and the filtering module;

the adding module is used for adding the suspected abnormal message to the cache region when the suspected abnormal message of the sliding operation is identified;

the third determining module is configured to determine a ranking weight of the suspected abnormal packet according to the first attribute feature and the second attribute feature of the suspected abnormal packet;

the filtering module is configured to, when the number of the suspected abnormal messages in the cache area exceeds a first set number, filter the suspected abnormal messages according to the sorting weight of the suspected abnormal messages in the cache area, and retain the suspected abnormal messages of the first set number.

15. The apparatus of claim 14, further comprising: a selection module and an aggregation module;

the selection module is configured to, for multiple statistical cycles corresponding to any attribute information, respectively select, according to a ranking weight of abnormal messages for the attribute information in each statistical cycle, a second set number of fifth messages from the abnormal messages for the attribute information in the statistical cycle, where the fifth messages include at least one abnormal message;

the aggregation module is configured to perform attribute aggregation on the same abnormal packet in the fifth packet selected from the multiple statistical periods to obtain aggregation information of each abnormal packet in the multiple statistical periods.

16. The apparatus of claim 15, wherein the aggregation module is specifically configured to:

17. A message processing apparatus, comprising: a memory and a processor;

the memory for storing a computer program;

the processor, coupled with the memory, to execute the computer program to: receiving one or more first messages, wherein the first messages have one or more attribute information; performing sliding operation on the sliding window according to any attribute information, and performing attribute information statistics on a second message in a current statistics period to determine a first attribute feature of the second message corresponding to the attribute information in each sliding window in the current statistics period and a second attribute feature of the second message corresponding to the attribute information in each sliding period corresponding to the sliding window; the second message is one or more first messages in the current statistical period; and determining the abnormal degree of the second message in the current statistical period according to the first attribute characteristic and the second attribute characteristic.

18. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 8.