CN109344170B

CN109344170B - Stream data processing method, system, electronic device and readable storage medium

Info

Publication number: CN109344170B
Application number: CN201811028222.9A
Authority: CN
Inventors: 赵坤
Original assignee: Advanced New Technologies Co Ltd
Current assignee: Advanced New Technologies Co Ltd; Advantageous New Technologies Co Ltd
Priority date: 2018-09-04
Filing date: 2018-09-04
Publication date: 2022-04-12
Anticipated expiration: 2038-09-04
Also published as: CN109344170A

Abstract

The embodiment of the specification provides a stream data processing method, which includes converting messages with corresponding identification information into events containing element information in a conversion rule through the conversion rule configured in advance to form an event stream, processing the event stream according to a processing logic configured in advance, and managing a processing result according to a result management rule configured in advance, so that a stream data processing process is effectively simplified.

Description

Stream data processing method, system, electronic device and readable storage medium

Technical Field

The embodiments of the present disclosure relate to the field of data processing technologies, and in particular, to a method and a system for processing stream data, an electronic device, and a readable storage medium.

Background

At present, with the development of services, especially the risk prevention and control scenario, the requirements for real-time defense and attack are higher and higher, so that the processing and query of real-time stream data become more and more urgent. However, the existing streaming data processing products basically require users to understand the compiling language or the processing framework, and the entry cost is high. Therefore, a simple and easy stream data processing scheme is required.

Disclosure of Invention

The embodiment of the specification provides a streaming data processing method, a streaming data processing system, an electronic device and a readable storage medium.

In a first aspect, an embodiment of the present specification provides a stream data processing method, including: receiving a message stream, wherein the message stream comprises a plurality of messages, and each message comprises identification information; searching a conversion rule associated with the identification information in a preset rule information base according to the identification information of each message, wherein the association information between the identification information and the conversion rule is stored in the rule information base, and the conversion rule comprises element information and processing logic; converting the corresponding message into an event containing the element information according to the processing logic in the searched conversion rule to form a corresponding event stream; and processing the event stream according to a preset processing logic to obtain a processing result, and managing the processing result according to a preset result management rule.

In a second aspect, an embodiment of the present specification provides a stream data processing system, including: the device comprises a receiving module, a searching module, a converting module and a processing module. A receiving module, configured to receive a message stream, where the message stream includes a plurality of messages, and each of the messages includes identification information. The searching module is used for searching a conversion rule associated with the identification information in a preset rule information base according to the identification information of each message, wherein the rule information base stores the associated information between the identification information and the conversion rule, and the conversion rule comprises element information and processing logic. And the conversion module is used for converting the corresponding message into the event containing the element information according to the processing logic in the searched conversion rule to form a corresponding event stream. And the processing module is used for processing the event stream according to a preset processing logic to obtain a processing result and managing the processing result according to a preset result management rule.

In a third aspect, an embodiment of the present specification provides an electronic device, including: a memory; one or more processors; and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the streaming data processing method provided by the first aspect when executing the program.

In a fourth aspect, the present specification provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the streaming data processing method provided in the first aspect.

The embodiment of the specification has the following beneficial effects:

in the stream data processing method provided in the embodiment of the present specification, the messages with the corresponding identification information are all converted into the event including the element information in the conversion rule through the conversion rule configured in advance to form the event stream, then the event stream is processed according to the processing logic configured in advance, and the processing result is managed according to the result management rule configured in advance. The stream data processing process is divided into a process of converting the message into the event and an event stream processing process, and the message is converted into the event according to the general conversion rule, so that the stream data processing process is simplified. Meanwhile, the decoupling of the message processing process and the streaming processing process is realized, the configuration is convenient, and the maintainability of the processing process is improved.

Drawings

Fig. 1 is a schematic view of an application scenario of an embodiment of the present disclosure;

fig. 2 is a flowchart of a stream data processing method provided in a first aspect of an embodiment of the present specification;

fig. 3 is a schematic structural diagram of a stream data processing system provided in a second aspect of an embodiment of the present specification;

fig. 4 is a schematic structural diagram of an electronic device provided in a third aspect of an embodiment of the present specification.

Detailed Description

In order to better understand the technical solutions, the technical solutions of the embodiments of the present specification are described in detail below with reference to the drawings and specific embodiments, and it should be understood that the specific features of the embodiments and embodiments of the present specification are detailed descriptions of the technical solutions of the embodiments of the present specification, and are not limitations of the technical solutions of the present specification, and the technical features of the embodiments and embodiments of the present specification may be combined with each other without conflict.

Herein, the term "and/or" is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In the embodiments of the present specification, stream data refers to a set of sequential, massive, fast, and continuous data sequences, and in general, a stream of data may be regarded as a dynamic data set that grows infinitely with time.

Fig. 1 is a schematic diagram of an operating environment suitable for the streaming data processing method provided in the embodiments of the present specification. As shown in fig. 1, one or more user terminals 100 (only one shown in fig. 1) may be connected via a network 200 to one or more servers 300 (only one shown in fig. 1) for data communication or interaction. The user terminal 100 may be a Personal Computer (PC), a notebook computer, a tablet computer, a smart phone, an electronic reader, a vehicle-mounted device, a network television, a wearable device, or other intelligent devices with network functions.

In the embodiment of the present specification, the user terminal 100 is installed with a client corresponding to a Server (Server) terminal to provide a service for a user. The client installed on the user terminal 100 may be an APP or a website for implementing services based on the internet, and provides an operation interface for various services for the user, and various operations performed by the user in the client, such as user click, login, transaction creation, payment, account transfer, etc., are converted into messages and sent to the server. The server is used for processing the message flow received in real time by a pre-configured processing method to obtain and manage the required processing result. The processing method in the server is configured by business personnel through the front-end business processing system, and the business personnel can configure and adjust the processing method in the front-end business processing system according to actual needs.

In a first aspect, an embodiment of the present specification provides a stream data processing method, which is applied to a server side, please refer to fig. 2, and the method may include at least the following steps S201 to S204. The steps S201 to S203 are used to convert the message stream into a corresponding event stream, and the step S204 is used to process the event stream.

Step S201, receiving a message stream, wherein the message stream comprises a plurality of messages, and each message comprises identification information;

various business operations such as clicking, logging in, creating transaction, payment, transferring and the like executed by the user at each application system client correspondingly generate messages, and the messages can be added into corresponding message queues and transmitted to the server. The message queue is a container for storing messages in the message transmission process. For example, if a user performs a transfer operation through a client, the client may generate a message corresponding to the transfer. In practical applications, the message may originate from AntQ, MetaQ, TT logs or message centers, etc.

The server collects messages generated by each user at each application system client to obtain message streams corresponding to different service scenes, namely different types of message streams, so that service personnel can analyze the messages, such as risk analysis. The messages under different service scenarios contain different identification information. Specifically, the identification information of the message may be a specific field in the message, which is used to characterize the service scenario corresponding to the message. For example, the type of message may include a user login message, a transfer message, a transaction message, a marketing campaign message, etc. according to different service scenarios.

It can be understood that the message is an underlying script file and is directed to a technician, and the technical threshold is high due to the difficulty in understanding by the service personnel. Therefore, in the embodiment of the present specification, the message stream is further converted into an event stream that can be understood by the service staff through the following steps S202 and S203, so as to prepare for the subsequent streaming data processing.

Step S202, according to the identification information of each message, searching the conversion rule associated with the identification information in the rule information base configured in advance.

The rule information base stores association information between identification information and conversion rules, and the conversion rules comprise element information and processing logic. It should be noted that, before executing the step S202, the service personnel is required to configure the rule information base in advance through the front-end service processing system according to the actual service scenario, that is, configure the event, that is, configure the conversion rule corresponding to the event and the identification information of the message corresponding to the event, and add the association information between the configured conversion rule and the identification information to the rule information base, so that the conversion rule of one or more events corresponding to the service scenario and the association information between the identification information of corresponding messages may be stored in the rule information base in advance, and may be configured specifically according to the requirements of the service personnel.

It should be noted that each conversion rule may be associated with identification information of a message in one service scenario, or may be associated with identification information of messages in multiple service scenarios.

Specifically, the conversion rule corresponding to the event mainly includes two elements, namely element information and processing logic. The element information includes various elements, and is abstract information of a concrete service, which basically includes who and who does what at what time (including how and why). For example, element information of a transfer event may include, but is not limited to: account transfer party ID, payee party ID, transfer time, transfer amount, account transfer party address, payee party address, transfer equipment, fund channel and the like. Therefore, in the embodiments of the present specification, an event may be understood as an abstract business scenario with respect to a concrete business.

Since the message is from the log of the application system, the log is processed by the business logic, and the processing logic can be configured by a DAG (Directed Acyclic Graph). In particular, processing logic may include one or more of multi-stream join, data back-lookup, filtering, and map (mapping) among other processes.

In particular, multi-stream join processing refers to merging and converting multiple different types of message streams into one event. For example, in a specific application scenario, there are payment messages for paying an application and payment messages for an store, where the two payment messages are messages of different service scenarios, have different identification information, but may belong to a payment event, and at this time, the payment messages for the application and the payment messages for the store need to be merged, and both of the two payment messages are converted into the payment event.

The data back-check processing means that when the element information included in the event is a plurality of elements, if only one or more elements in the event can be analyzed from the message corresponding to the event, the one or more elements are queried in a pre-stored database to obtain elements not included in other messages, so that the information is completed. Specifically, when it is assumed that an event includes an element a, an element B, an element C, and an element D, when only the element a and the element B can be parsed from a message corresponding to the event, and the element C and the element D cannot be parsed, the element a and/or the element B may query the corresponding element C and element D in a database in which association information between the element a and/or the element B and the element C and the element D is stored in advance, thereby completing element information required by the event.

For example, in a specific application scenario, a certain type of message includes an ID of a user, and element information of an event corresponding to the message includes, in addition to the ID of the user, information that is not included in the message, such as the age and gender of the user.

The filtering process refers to filtering the message, for example, the message stream corresponding to a certain service scenario may be filtered according to a given requirement to obtain a message meeting the requirement, and then the filtered message is further converted into a corresponding event. For example, for the messages of the marketing campaigns, the messages of a plurality of marketing campaigns such as P1, P2, P3 and P4 are included, but the current business analysis requirement only needs to analyze the messages of the two campaigns P1 and P2, and then corresponding filtering rules can be set to screen the messages of the two campaigns P1 and P2 from the messages of the marketing campaigns.

map refers to mapping processing such as splitting, splicing, converting and the like of character strings in a message, and is specifically set according to actual needs. For example, the first character of a string in a message may be intercepted. For another example, some character strings or characters extracted from the message may be subjected to a concatenation process, and if the character a and the character b are concatenated, ab is obtained. As another example, a particular string in a message may be translated to a certain element name in a corresponding event.

It should be noted that, for each message received in step S201, when the conversion rule corresponding to the identification information of the message is found in the rule information base, the following step S203 is executed, and when the conversion rule corresponding to the identification information of the message is not found in the rule information base, the message is not subjected to event conversion, that is, the following step S203 is not executed.

Step S203, converting each message into an event including the element information according to the processing logic in the searched conversion rule, and forming a corresponding event stream.

In this embodiment of the present specification, one message may correspond to one event, or multiple messages may correspond to one event, and may be specifically defined according to a specific service scenario. For example, when the message includes a user login message, a transfer message, a transaction message, a marketing campaign message, accordingly, the event is also classified into a user login event, a transfer event, a transaction event, and the like.

And processing the message through the processing logic in the searched conversion rule, so that each element value in the element information included in the conversion rule can be obtained, and the corresponding event can be obtained.

It should be noted that, in order to monitor the event conversion process, timely find possible problems in the processing logic in each conversion rule, and ensure the reliability of the event conversion process, the event conversion process may be displayed in an event conversion window of the front-end service processing system, so as to realize visualization of the event conversion process, so that service personnel can monitor the event conversion process, and when there is a problem in the process, timely correct the corresponding conversion rule.

It can be understood that, if the above steps S202 to S203 are performed on the received message streams corresponding to different service scenarios transmitted by each application system client, the message streams corresponding to different service scenarios can be converted into event streams corresponding to different service scenarios, respectively. The message is converted into the event, so that multiplexing of the message can be realized, the corresponding event can be consumed for multiple times, for example, when risk analysis is carried out on a transfer service scene, the related transfer event can be used when the transfer times of the user in the past day and the transfer times of the user in the past month are counted. As an alternative, these event streams may be stored in a pre-constructed event library, so that the following step S204 is subsequently performed on these event streams according to the actual business analysis requirements.

And step S204, processing the event stream according to a preset processing logic to obtain a processing result, and managing the processing result according to a preset result management rule.

It will be appreciated that prior to processing the event stream, consideration needs to be given to whether the event stream available in the current system meets the current analysis requirements. When an event configured by the current system cannot meet an analysis requirement, for example, a service scene is newly added or a service scene to be analyzed is newly added, a corresponding newly added event needs to be configured, so that a corresponding event stream is obtained, and the analysis requirement is met. The configuration process of the newly added event comprises the following steps: the step of updating the rule information base specifically includes: acquiring a first configuration instruction, wherein the first configuration instruction comprises a conversion rule of a newly added event and identification information of a corresponding message; and associating the conversion rule corresponding to the newly added event with the identification information, and adding the associated information into the rule information base. Therefore, an event stream corresponding to the newly added event can be formed according to the event conversion process, and the analysis requirement is met. It should be noted that the configuration process of the newly added event needs to be executed before step S204, specifically, may be executed before step S201, or may be executed substantially simultaneously with step S201, step S202, or step S203.

In an embodiment of this specification, considering that event streams involved may be different according to different service analysis requirements, before processing the event streams according to actual service analysis requirements, the event streams needing to be processed at this time need to be screened. That is, the processing the event stream according to the preconfigured processing logic to obtain a processing result, and managing the processing result according to the preconfigured result management rule may specifically include: filtering the event stream according to a preset filtering condition to screen out the event stream to be processed; and processing the event stream to be processed according to a preset processing logic to obtain a processing result, and managing the processing result according to a preset result management rule. Of course, in other embodiments of the present description, according to a specific service analysis requirement, all event streams obtained in step S203 may also be directly processed to obtain an analysis result.

Wherein, the filtering condition can be configured according to actual needs. As an alternative, the filtering condition may be configured to screen out event streams that need to be involved in the analysis from a plurality of event streams currently obtained by the system, as event streams to be processed.

Specifically, the service personnel may pre-select an event stream required to be related to the analysis from event streams corresponding to different service scenarios according to actual analysis requirements, as an event stream to be processed, that is, before executing step S204, the service personnel needs to receive an event stream selection instruction, and determine the event stream to be processed according to the event stream selection instruction. For example, the service personnel configures M events in total, and through the above step S203, M event streams can be obtained and stored in the event library, and the service personnel can select N event streams from the event library in advance according to the actual analysis requirement to perform subsequent stream calculation. Wherein M and N are positive integers, and N is less than M.

For example, in a specific application scenario, the event stream obtained through step S203 includes a user login event stream, a transfer event stream, a transaction event stream, and a marketing event stream. If the service personnel only needs to analyze the transaction event stream at this time, the event stream needing to be processed can be selected in the event stream selection window of the front-end service processing system in advance, so that the server only processes the event stream selected by the service personnel when executing the step S204, and a processing result needed by the analysis is obtained.

Specifically, the manner for the service personnel to select the event stream to be involved in the analysis may be: the event stream selection window of the front-end service processing system shows all kinds of event streams which can be obtained by the current system, and service personnel can mark the event streams by clicking the event streams required by the analysis so as to generate corresponding event stream selection instructions. These marked event streams are the event streams that the business personnel select to be involved in the analysis. Or, the mode of the service personnel selecting the event stream to be related to the analysis may also be: business personnel can directly edit an event stream selection instruction through Structured Query Language (SQL), the event stream selection instruction comprises an event stream screening condition, and the event stream meeting the screening condition is the event stream selected by the business personnel and required to be related to the analysis.

In addition, it is considered that each service scenario may include a plurality of different service operations, and accordingly, for each event stream, the event stream may be further refined into a plurality of sub-event streams according to different specific element values, while in some service analysis requirements, only a specific sub-event stream may need to be screened from the event stream for analysis. Therefore, in an embodiment of the present specification, the filtering condition may be further configured to, after the event streams to be referred to in the analysis are screened out, re-screen out sub-event streams from the event streams as the event streams to be processed based on the specific element values for the screened out event streams.

For example, for a transfer scenario, the transfer scenario may include transfer from a treasury and transfer from a bank card, and accordingly, in the transfer event stream, the transfer event stream may be subdivided into a treasury transfer subevent stream and a transfer subevent stream from a bank card according to corresponding element values in the transfer event stream, such as a fund channel element value, and if the analysis only needs to analyze the treasury transfer event in the transfer event stream, the treasury transfer subevent stream needs to be screened out from the transfer event stream to be used as an event stream to be processed.

In step S204, the processing logic and the result management rule are configured in advance by the service personnel according to the actual analysis requirement before step S204 is executed. And the processing logic is used for processing the event stream to obtain a required processing result. For example, when a transaction scenario needs to be analyzed, the transaction times of a specific user in a preset time period, such as a past day or a past month, may be counted for a transaction event stream. The result management rule is used for managing the processing result, and may specify a target position where the result is desired to be loaded, or may directly output the processing result as a specific feature.

Specifically, the process logic configuration process may include: acquiring a second configuration instruction, wherein the second configuration instruction comprises a machining logic generated based on a trigger operation executed by a user on a preset machining logic configuration window; the processing logic is retrieved from the second configuration instructions.

According to actual analysis requirements, a user, namely a service person, can write event stream processing logic by using an SQL editor in a processing logic configuration window of the front-end service processing system, generate a second configuration instruction containing the processing logic, and send the second configuration instruction to the background server to complete the configuration of the processing logic. It can be understood that most business personnel have SQL work bases and can use standard SQL to compile event stream processing logic, so that the business personnel do not need to know new programming languages or background processing frames, and the business personnel can conveniently and quickly get on the home and easily enter the home.

In order to further simplify the operation of service personnel, a plurality of preset templates can be preset in the processing logic configuration window, and at the moment, the processing logic can be generated through the preset template triggered by a user and the modification instruction input by the user. Specifically, the service personnel can select an applicable template according to the actual service analysis requirement, and modify the template according to the actual requirement based on the SQL editor to obtain the corresponding event stream processing logic.

For example, according to different requirements for windows in different service scenarios, the following 4 Window processing templates are provided in GroupBy Window in SQL: Global-Global window, all data in one window; tumble-rolling window, the window data has fixed size, the window data does not overlap; the method comprises the steps of Hop-sliding window, wherein window data have fixed size and fixed window reconstruction frequency, and the window data are overlapped; Session-Session window, window data has no fixed size, the window is divided according to the window data activity degree, and the window data has no superposition. The service personnel can select the applicable window processing template according to the actual requirement.

Specifically, the result management rule configuration process may include: acquiring a third configuration instruction, wherein the third configuration instruction comprises a result management rule, and the result management rule comprises the steps of storing the processing result to a specified target position according to a preset format and/or outputting the processing result according to the preset format; and acquiring the result management rule from the third configuration instruction.

The target position may be a position such as HDFS (Hadoop Distributed File System), Hbase, or geobase.

The preset format can be set according to specific business analysis needs. For example, in a specific application scenario, the configured processing logic is specifically configured to count the number of transactions per hour for a specific user of a certain application within a specific time period, and the preset format may include a user ID, a time node, and a corresponding number of transactions. Assume that the particular users include user 1, user 2, and user 3, and the particular time period is 11 of 8 months and 29 days in 2018: 00 to 14: 00, the processing result can be stored or output to the front-end service processing system according to table 1.

TABLE 1

In table 1, Num11, Num21, and Num31 respectively correspond to 11: 00 to 12: 00, transaction times of user 1, user 2, and user 3; num12, Num22, and Num32 correspond to 12: 00 to 13: 00, transaction times of user 1, user 2, and user 3; num13, Num23, and Num33 correspond to representations 13: 00 to 14: 00, user 1, user 2, and user 3.

It should be noted that, in the embodiment of the present specification, in order to ensure that the preconfigured rule information base, the processing logic, and the result management rule can meet the expectations of business personnel, before the preconfigured rule information base, the processing logic, and the result management rule are brought online, a test step may be performed. The testing steps specifically include: and acquiring a preset number of sample messages, and processing the sample messages according to the sample messages in the steps S202 to S204 based on a preset rule information base, a preset processing logic and a preset result management rule to obtain a test result. The service personnel can judge whether the test result meets the expected requirement, if so, the rule information base, the processing logic and the result management rule which are configured in advance are put on line formally to carry out actual flow data processing, and if not, the corresponding logic or rule needs to be corrected.

In addition, in an embodiment of the present disclosure, in order to improve the reliability of the present solution, a monitoring alarm function may be further provided. The monitoring alarm function may include a monitoring alarm for a data source, i.e., the message stream received in step S201, and/or a monitoring alarm for a processing result obtained in step S204.

Specifically, the monitoring alarm process for the data source may include: monitoring a specific parameter value in a specific message in a received message stream, matching the parameter value with a preset standard threshold value, if the parameter value is not matched with the preset standard threshold value, indicating that the message is an abnormal message, and alarming the abnormal message so that a service worker can process the abnormal message in time. The specific parameter value can be selected according to actual needs, and the preset standard threshold value is set according to the selected specific parameter value. For example, the particular parameter value may be a bank card number, an identification number, or the like. Correspondingly, the preset standard threshold may be a standard digit of a bank card number, a standard digit of an identification number, and the like.

Specifically, the process of monitoring and alarming the processing result may include: inputting the processing result into a monitoring model corresponding to the processing logic to obtain a detection result, wherein the monitoring model is an unsupervised model which is trained in advance through historical processing result data; and outputting preset alarm information when the detection result meets the preset alarm condition.

For example, for a certain processing logic of the transfer event stream, the transfer times of each user in a specified time period are counted, and if unsupervised training is performed according to historical statistical result data, and it is obtained that 90% of users in a certain type of users have transfer times within 30 times in a day, alarm conditions corresponding to the type of users may be set according to the training result, and if: in this type of user, an alert is given when more than 50% of the users have more than 60 transfers during the day. At this time, if the currently obtained processing result is input into the monitoring model, and 60% of users in the type of users have more than 60 transfer times in one day, preset alarm information is output to prompt related personnel to carry out risk prevention and control in time.

The stream data processing method provided in the embodiment of the present specification divides a stream data processing process into a process of converting a message into an event and an event stream processing process, and converts the message into the event according to a general conversion rule, thereby simplifying the stream data processing process. Meanwhile, the decoupling of the message processing process and the streaming processing process is realized, the configuration is convenient, and the maintainability of the processing process is improved.

In addition, the stream data processing method provided in the embodiments of the present specification deposits the message conversion link according to the service knowledge that can be understood by the service, so as to implement multiplexing of the message, and implement the purpose of one-time message access and global service multiplexing. The processing logic is configured by using the standard SQL, so that business personnel do not need to know a new programming language or a background processing frame, and a set of fixed templates are abstracted according to the business logic, so that the business personnel can quickly and easily enter the door. In addition, a visual running function is provided, so that a user can immediately see the accumulated result after finishing accumulation, and the corresponding logic or rule is corrected before formal online, and the accuracy of the processing result is ensured.

In a second aspect, based on the same inventive concept, an embodiment of the present specification further provides a stream data processing system 30, please refer to fig. 3, including:

a receiving module 31, configured to receive a message stream, where the message stream includes a plurality of messages, and each of the messages includes identification information;

a searching module 32, configured to search, according to the identification information of each message, a conversion rule associated with the identification information in a preconfigured rule information base, where the rule information base stores association information between the identification information and the conversion rule, and the conversion rule includes element information and processing logic;

a conversion module 33, configured to convert the corresponding message into an event including the element information according to the processing logic in the searched conversion rule, and form a corresponding event stream;

and the processing module 34 is configured to process the event stream according to a preconfigured processing logic to obtain a processing result, and manage the processing result according to a preconfigured result management rule.

As an optional embodiment, the system further includes: a first configuration module to: acquiring a first configuration instruction, wherein the first configuration instruction comprises a conversion rule of a newly added event and identification information of a corresponding message; and associating the conversion rule corresponding to the newly added event with the identification information, and adding the associated information into the rule information base.

As an optional embodiment, the system further includes: a second configuration module to: acquiring a second configuration instruction, wherein the second configuration instruction comprises a machining logic generated based on a trigger operation executed by a user on a preset machining logic configuration window; the processing logic is retrieved from the second configuration instructions.

As an optional embodiment, a plurality of preset templates are set in the machining logic configuration window, and the machining logic is generated through the preset template triggered by the user and the modification instruction input by the user.

As an alternative embodiment, the processing module 34 comprises: the filtering submodule 341 is configured to filter the event stream according to a preset filtering condition, and screen out an event stream to be processed; the processing submodule 342 is configured to process the event stream to be processed according to a preconfigured processing logic to obtain a processing result, and manage the processing result according to a preconfigured result management rule.

As an optional embodiment, the system further includes: a third configuration module to: acquiring a third configuration instruction, wherein the third configuration instruction comprises a result management rule, and the result management rule comprises the steps of storing the processing result to a specified target position according to a preset format and/or outputting the processing result according to the preset format; and acquiring the result management rule from the third configuration instruction.

As an optional embodiment, the system further includes: the monitoring alarm module is used for: inputting the processing result into a monitoring model corresponding to the processing logic to obtain a detection result, wherein the monitoring model is an unsupervised model which is trained in advance through historical processing result data; and outputting preset alarm information when the detection result meets the preset alarm condition.

It should be noted that, the embodiment of the present specification provides the stream data processing system 30, wherein the specific manner in which each module performs the operation has been described in detail in the above method embodiment, and will not be described in detail here.

In a third aspect, based on the same inventive concept as the streaming data processing method in the foregoing embodiments, the present invention further provides an electronic device, as shown in fig. 4, including a memory 404, one or more processors 402, and a computer program stored on the memory 404 and executable on the processor 402, wherein the processor 402 executes the computer program to implement the steps of the streaming data processing method.

Where in fig. 4 a bus architecture (represented by bus 400) is shown, bus 400 may include any number of interconnected buses and bridges, and bus 400 links together various circuits including one or more processors, represented by processor 402, and memory, represented by memory 404. The bus 400 may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface 405 provides an interface between the bus 400 and the receiver 401 and transmitter 403. The receiver 401 and the transmitter 403 may be the same element, i.e., a transceiver, providing a means for communicating with various other apparatus over a transmission medium. The processor 402 is responsible for managing the bus 400 and general processing, while the memory 404 may be used for storing data used by the processor 402 in performing operations.

It will be appreciated that the configuration shown in fig. 4 is merely illustrative and that the electronic device described above may also include more or fewer components than shown in fig. 4, or have a different configuration than shown in fig. 4. The components shown in fig. 4 may be implemented in hardware, software, or a combination thereof.

In a fourth aspect, based on the inventive concept of the stream data processing method in the foregoing embodiments, the present invention also provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of any one of the foregoing stream data processing methods.

The description has been presented with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the description. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present specification have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all changes and modifications that fall within the scope of the specification.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present specification without departing from the spirit and scope of the specification. Thus, if such modifications and variations of the present specification fall within the scope of the claims of the present specification and their equivalents, the specification is intended to include such modifications and variations.

Claims

1. A streaming data processing method, comprising:

receiving a message stream, wherein the message stream comprises a plurality of messages, and each message comprises identification information; the identification information of the message is used for representing the service scene corresponding to the message, and the message under different service scenes contains different identification information;

searching a conversion rule associated with the identification information in a preset rule information base according to the identification information of each message, wherein the association information between the identification information and the conversion rule is stored in the rule information base, and the conversion rule comprises element information and processing logic; the processing logic comprises one or more of multi-stream join, data reverse lookup, filtering, and map processing;

converting the corresponding message into an event containing the element information according to the processing logic in the searched conversion rule to form a corresponding event stream;

processing the event stream according to a preset processing logic to obtain a processing result, and managing the processing result according to a preset result management rule; the processing logic of the event stream is generated by writing in a processing logic configuration window by a user, or generated by a preset template in the processing logic configuration window triggered by the user and a modification instruction input by the user in the processing logic configuration window.

2. The method of claim 1, further comprising: updating the rule information base specifically includes:

acquiring a first configuration instruction, wherein the first configuration instruction comprises a conversion rule of a newly added event and identification information of a corresponding message;

and associating the conversion rule corresponding to the newly added event with the identification information, and adding the associated information into the rule information base.

3. The method of claim 1, wherein before managing the processing result according to the preconfigured result management rule, the method further comprises:

acquiring a second configuration instruction, wherein the second configuration instruction comprises a machining logic generated based on a trigger operation executed by a user on a preset machining logic configuration window;

the processing logic is retrieved from the second configuration instructions.

4. The method of claim 3, wherein a plurality of preset templates are disposed in the tooling logic configuration window.

5. The method of claim 1, wherein processing the event stream according to a preconfigured processing logic to obtain a processing result, and managing the processing result according to a preconfigured result management rule comprises:

filtering the event stream according to a preset filtering condition to screen out the event stream to be processed;

and processing the event stream to be processed according to a preset processing logic to obtain a processing result, and managing the processing result according to a preset result management rule.

6. The method of claim 1, wherein before processing the event stream according to the preconfigured processing logic to obtain a processing result and managing the processing result according to the preconfigured result management rule, the method further comprises:

acquiring a third configuration instruction, wherein the third configuration instruction comprises a result management rule, and the result management rule comprises the steps of storing the processing result to a specified target position according to a preset format and/or outputting the processing result according to the preset format;

and acquiring the result management rule from the third configuration instruction.

7. The method of claim 1, wherein processing the event stream according to the pre-configured processing logic further comprises, after obtaining a processing result:

inputting the processing result into a monitoring model corresponding to the processing logic to obtain a detection result, wherein the monitoring model is an unsupervised model which is trained in advance through historical processing result data;

and outputting preset alarm information when the detection result meets the preset alarm condition.

8. A streaming data processing system, comprising:

a receiving module, configured to receive a message stream, where the message stream includes a plurality of messages, and each of the messages includes identification information; the identification information of the message is used for representing the service scene corresponding to the message, and the message under different service scenes contains different identification information;

the searching module is used for searching a conversion rule associated with the identification information in a preset rule information base according to the identification information of each message, wherein the rule information base stores the associated information between the identification information and the conversion rule, and the conversion rule comprises element information and processing logic; the processing logic comprises one or more of multi-stream join, data reverse lookup, filtering, and map processing;

the conversion module is used for converting the corresponding message into an event containing the element information according to the processing logic in the searched conversion rule to form a corresponding event stream;

the processing module is used for processing the event stream according to a preset processing logic to obtain a processing result and managing the processing result according to a preset result management rule; the processing logic of the event stream is generated by writing in a processing logic configuration window by a user, or generated by a preset template in the processing logic configuration window triggered by the user and a modification instruction input by the user in the processing logic configuration window.

9. The system of claim 8, further comprising: a first configuration module to:

10. The system of claim 8, further comprising: a second configuration module to:

the processing logic is retrieved from the second configuration instructions.

11. The system of claim 10, wherein a plurality of preset templates are disposed in the finishing logic configuration window.

12. The system of claim 8, the processing module comprising:

the screening submodule is used for filtering the event stream according to a preset filtering condition to screen out the event stream to be processed;

and the processing submodule is used for processing the event stream to be processed according to the preset processing logic to obtain a processing result and managing the processing result according to a preset result management rule.

13. The system of claim 8, further comprising: a third configuration module to:

14. The system of claim 8, further comprising: the monitoring alarm module is used for:

15. An electronic device, comprising:

a memory;

one or more processors; and

a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of any one of claims 1-7 when executing the program.

16. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.