CN115984001A

CN115984001A - Event stream processing method, event stream processing device, electronic device, medium, and program product

Info

Publication number: CN115984001A
Application number: CN202310137576.1A
Authority: CN
Inventors: 曾胜泓
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2023-02-20
Filing date: 2023-02-20
Publication date: 2023-04-18

Abstract

The disclosure provides an event stream processing method which can be used in the technical field of big data. The event stream processing method comprises the following steps: collecting original data from m data source systems, wherein the original data has a collection time stamp, and m is an integer greater than or equal to 1; aggregating the original data in a preset time window according to the acquisition timestamp based on request information of a terminal data application system to obtain a window event, wherein the request information is a specific requirement for aggregating the original data; extracting the window event according to a set time period to obtain an extraction result; and transmitting the extraction result to a storage library for extraction by the terminal data application system. The disclosure also includes an event stream processing apparatus, an electronic device, a medium, and a computer program product.

Description

Event stream processing method, device, electronic equipment, medium and program product

Technical Field

The present disclosure relates to the field of big data technologies, and in particular, to an event stream processing method, apparatus, electronic device, medium, and computer program product.

Background

The existing event processing methods are generally two, namely stream processing and batch processing.

Among them, streaming processing is streaming processing, which assumes that the potential value of data is the freshness of the data and requires processing as soon as possible to obtain the result. In this way, the data arrives in a stream, and during the continuous arrival of the data, only a small portion of the stream data is stored in a limited memory because the stream carries a large amount of data. Streaming is typically used for online applications, operating on the order of seconds or milliseconds.

The data are firstly stored and then analyzed, the data are firstly divided into a plurality of data blocks, then the data blocks are processed in parallel and generate intermediate results in a distributed mode, and finally the intermediate results are combined to generate a final result.

In the two event processing modes, because the data volume is large, a large amount of cpu resources, memory resources and network resources are required to be occupied during transmission, conversion and calculation, and therefore, the calculation resources, the memory resources and the network resources are consumed.

Disclosure of Invention

In view of the above, the present disclosure provides an event stream processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that save computing resources, memory resources, and network resources.

One aspect of the present disclosure provides an event stream processing method, including: collecting original data from m data source systems, wherein the original data has a collection time stamp, and m is an integer greater than or equal to 1; aggregating the original data in a preset time window according to the acquisition timestamp based on request information of a terminal data application system to obtain a window event, wherein the request information is a specific requirement for aggregating the original data; extracting the window event according to a set time period to obtain an extraction result; and transmitting the extraction result to a storage library for extraction by the terminal data application system.

According to the event stream processing method of the embodiment of the disclosure, based on the request information of the terminal data application system, the original data in the preset time window is aggregated according to the acquisition time stamp to obtain the window event, and the window event is extracted according to the set time period to obtain the extraction result. The use requirement of the terminal data application system can be met only by transmitting the extraction result to the storage library, so that the original data before aggregation does not need to be transmitted, and the consumption of network resources and memory resources is directly reduced; in addition, because the target expressions of the window events in each time window are the same, the window events can be merged across the time windows under the condition of not amplifying errors to obtain an extraction result, and a summarizing effect is obtained under the condition of not losing accuracy, so that the memory amount required by calculation per second can be reduced, the internal memory consumption can be reduced to the maximum extent, the increment calculation amount is reduced, the calculation resources are saved, and the relative accuracy of the aggregation module with the aggregation function is further maintained.

In some embodiments, the collecting raw data from each data source system comprises: setting a data interface device for each data source system, formulating m external protocols, wherein the m external protocols correspond to the m data source systems one by one, and acquiring different types of data generated by the m data source systems to a data flow pipeline through the m external protocols, wherein the event flow processing method is realized in the data flow pipeline.

In some embodiments, before aggregating the original data in a preset time window according to the acquisition timestamp to obtain a window event based on the request information of the terminal data application system, the event stream processing method further includes: integrating the original data according to a pre-configured data structure to obtain integrated data, wherein the pre-configured data structure comprises a data format and/or a data field; the acquiring method comprises the following steps that the request information based on the terminal data application system aggregates the original data in a preset time window according to the acquisition time stamp to obtain a window event, and comprises the following steps: and aggregating the integrated data in a preset time window according to the acquisition time stamp based on the request information of the terminal data application system to obtain a window event.

In some embodiments, the aggregating, based on the request information of the terminal data application system, the integrated data in a preset time window according to the collection timestamp to obtain a window event includes: and calculating the integration data in a preset time window by using an aggregation algorithm according to the acquisition time stamp based on the request information of the terminal data application system to obtain a window event.

In some embodiments, the transmitting the extraction result to a repository for extraction by the terminal data application system includes: and queuing and transmitting the extraction results to a storage library in a multithreading mode for the terminal data application system to extract.

In some embodiments, before the transmitting the extraction result to the repository for the terminal data application system to extract, the event stream processing method further includes: and carrying out formatting conversion on the extraction result according to a data format required by the terminal data application system: the transmitting the extraction result to a storage library for the terminal data application system to extract comprises: and transmitting the extraction result after the formatting conversion to a storage library for the extraction of the terminal data application system.

Another aspect of the present disclosure provides an event stream processing apparatus including: the device comprises a collecting device and a processing device, wherein the collecting device is used for collecting original data from m data source systems, the original data has a collecting time stamp, and m is an integer greater than or equal to 1; the aggregation device is used for executing request information based on a terminal data application system, aggregating the original data in a preset time window according to the acquisition timestamp to obtain a window event, wherein the request information is a specific requirement for aggregating the original data; the extraction device is used for extracting the window event according to a set time period to obtain an extraction result; and the transmission device is used for transmitting the extraction result to a storage library for the extraction of the terminal data application system.

Another aspect of the present disclosure provides an electronic device comprising one or more processors and one or more memories, wherein the memories are configured to store executable instructions that, when executed by the processors, implement the method as described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.

Another aspect of the disclosure provides a computer program product comprising a computer program comprising computer executable instructions for implementing the method as described above when executed.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1 schematically illustrates an exemplary system architecture to which the methods, apparatus, according to some embodiments of the present disclosure, may be applied;

FIG. 2 schematically illustrates a flow diagram of an event stream processing method according to some embodiments of the present disclosure;

FIG. 3 schematically illustrates a flow diagram of an event stream processing method according to some embodiments of the present disclosure;

fig. 4 schematically illustrates a flow chart for aggregating integrated data within a predetermined time window according to a collection timestamp to obtain a window event based on request information of a terminal data application system according to some embodiments of the present disclosure;

FIG. 5 schematically illustrates a flow diagram of transferring extraction results into a repository for extraction by a terminal data application system, according to some embodiments of the present disclosure;

FIG. 6 schematically illustrates a flow diagram of an event stream processing method according to some embodiments of the present disclosure;

fig. 7 schematically shows a block diagram of an event stream processing apparatus according to an embodiment of the present disclosure;

fig. 8 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, necessary security measures are taken, and the customs of the public order is not violated. In the technical scheme of the disclosure, the data acquisition, collection, storage, use, processing, transmission, provision, disclosure, application and other processing are all in accordance with the regulations of relevant laws and regulations, necessary security measures are taken, and the public order and good custom are not violated.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include, but not be limited to, systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). The terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features.

Among them, streaming processing is streaming processing, which assumes that the potential value of data is the freshness of the data and requires processing as soon as possible to obtain the result. In this way, the data arrives in a stream, and during the continuous arrival of the data, only a small portion of the stream data is stored in a limited memory because the stream carries a large amount of data. Streaming is typically used for online applications, working on the order of seconds or milliseconds.

In the two event processing modes, because the data volume is large, a large amount of computing resources, memory resources and network resources are required to be occupied during transmission, conversion and calculation, and therefore the computing resources, the memory resources and the network resources are consumed.

Embodiments of the present disclosure provide an event stream processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. The event stream processing method comprises the following steps: acquiring original data from m data source systems, wherein the original data has an acquisition time stamp, and m is an integer greater than or equal to 1; aggregating original data in a preset time window according to a collecting time stamp based on request information of a terminal data application system to obtain a window event, wherein the request information is a specific requirement for aggregating the original data; extracting window events according to a set time period to obtain an extraction result; and transmitting the extraction result to a storage library for extraction by the terminal data application system.

It should be noted that the event stream processing method, apparatus, electronic device, computer-readable storage medium, and computer program product of the present disclosure may be used in the field of big data technology, and may also be used in any field other than the field of big data technology, such as the financial field, and the field of the present disclosure is not limited herein.

Fig. 1 schematically illustrates an exemplary system architecture 100 to which the event stream processing method, apparatus, electronic device, computer-readable storage medium and computer program product may be applied, according to embodiments of the disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may use

terminal devices

101, 102, 103 to interact with a server 105 over a network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the event stream processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the event stream processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the server 105. The event stream processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Correspondingly, the event stream processing apparatus provided in the embodiment of the present disclosure may also be disposed in a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The event stream processing method according to the embodiment of the present disclosure will be described in detail below with reference to fig. 2 to 6 based on the scenario described in fig. 1.

Fig. 2 schematically shows a flow chart of an event stream processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the event stream processing method of this embodiment includes operations S210 to S240.

In operation S210, raw data from m data source systems is acquired, wherein the raw data has an acquisition time stamp, and m is an integer greater than or equal to 1.

As a possible implementation manner, operation S210 collects raw data from various data source systems, including operation S211.

In operation S211, a data interface is set for each data source system, m external protocols are formulated, the m external protocols correspond to the m data source systems one to one, and different types of data generated by the m data source systems are collected to a data stream pipeline through the m external protocols, wherein the event stream processing method is implemented in the data stream pipeline. Therefore, different types of data can be collected, and the collection requirements of various types of data are met. The acquisition of raw data from various data source systems may be facilitated by operation S211.

In operation S220, original data in a preset time window is aggregated according to a collection timestamp based on request information of the terminal data application system to obtain a window event, where the request information is a specific requirement for aggregating the original data.

As a possible implementation manner, as shown in fig. 3, before the operation S220 aggregates the original data in the preset time window according to the collection timestamp based on the request information of the terminal data application system to obtain the window event, the event stream processing method further includes an operation S001.

In operation S001, the original data is integrated according to a preconfigured data structure to obtain integrated data, where the preconfigured data structure includes a data format and/or a data field.

It can be understood that, the data interface device collects data of the data stream pipeline, the collected data types are inconsistent due to different data sources, for example, there are structural data such as json, map and list, and there are unstructured data such as irregular character strings, but to extract useful information from the data in the data stream pipeline, it is necessary to process the unstructured data, for example, add, modify or delete data fields to convert the unstructured data into structured data. The preconfigured data structure may also include filling rules for field values, such as adding, modifying, or deleting content under data fields according to the filling rules. The data format of the collected data can also be converted according to a preconfigured data format, so that a desired data pattern is obtained through the preconfigured data structure, and the unstructured data is converted.

Specifically, when each data interface device collects original data to a data stream pipeline, information of the data interface device is added and inserted into the collected data; or the data is converted into a data format when being transmitted into a data stream pipeline, and the data format is converted into a format expected by us; if the data redundant information is too much, the data can be deleted, and the circulation of the junk data in a data flow pipeline is reduced; for unstructured data, data fields in a template of a pre-configured data structure are acquired in a data stream pipeline through the pre-configured data structure and converted into formatted data. For example, the data type of the preset template is json, data fields are defined in the json, values are obtained by matching the data fields in unstructured character strings through regular expressions according to the defined data fields, and then the values are filled in the json of the preset template, so that data conversion of the type is completed.

The preprocessing of the original data can be facilitated through the operation S001, and the processed integrated data can be conveniently applied subsequently.

Therefore, the operation S220 may include aggregating the raw data within the preset time window according to the collection timestamp based on the request information of the terminal data application system to obtain the window event, which may include an operation S221.

In operation S221, based on the request information of the terminal data application system, the integrated data in the preset time window is aggregated according to the acquisition timestamp, so as to obtain a window event.

For example, a time window may be preset as t seconds as required, the integrated data within t seconds may be aggregated according to the time sequence of collecting the timestamps, the aggregation mode may be performed according to a required mode in the request information, and a window event may be formed after the integrated data within t seconds is aggregated. Through operation S221, the request information based on the terminal data application system can be conveniently implemented, and the original data in the preset time window is aggregated according to the collection timestamp, so as to obtain the window event.

In some specific examples, as shown in fig. 4, the operation S221, based on the request information of the terminal data application system, aggregates the integrated data within the preset time window according to the collection timestamp to obtain the window event, including operation S2211.

In operation S2211, based on the request information of the terminal data application system, the aggregation algorithm is used to calculate the integration data in the preset time window according to the collection timestamp, so as to obtain a window event. Here, the aggregation algorithm may include a SUM algorithm (SUM function), an averaging algorithm (AVG function), a maximum value algorithm (MAX function), a minimum value algorithm (MIN function), and a COUNT algorithm (COUNT function).

The integration data in the preset time window is calculated by using an aggregation algorithm according to the acquisition time stamp, and can be understood as the integration data in the preset time window is calculated by using any one of a summation algorithm, an averaging algorithm, a maximum value algorithm, a minimum value algorithm and a counting algorithm according to the acquisition time stamp; the method can also be understood as calculating the integrated data in a preset time window according to the acquisition time stamp by using any two of a summation algorithm, an averaging algorithm, a maximum value algorithm, a minimum value algorithm and a counting algorithm; the method can also be understood as calculating the integrated data in a preset time window according to the acquisition time stamp by using any three of a summation algorithm, an averaging algorithm, a maximum value algorithm, a minimum value algorithm and a counting algorithm; the method can also be understood as calculating the integrated data in a preset time window according to the acquisition time stamp by using any four of a summation algorithm, an averaging algorithm, a maximum value algorithm, a minimum value algorithm and a counting algorithm; it can also be understood that the integration data within the predetermined time window is calculated according to the acquisition time stamp using a summation algorithm, an averaging algorithm, a maximum algorithm, a minimum algorithm, and a counting algorithm.

Through operation S2211, the request information based on the terminal data application system may be conveniently implemented, and the integrated data in the preset time window is aggregated according to the collection timestamp, so as to obtain the window event.

In operation S230, the window event is extracted according to the set time period, resulting in an extraction result. For example, the set time period may be set to T seconds as needed, and the T seconds include T/T time windows, so that T/T window events may be extracted according to the set time period to form one extraction result.

In operation S240, the extraction result is transmitted to a repository for the terminal data application system to extract.

According to some embodiments of the present disclosure, as shown in fig. 5, operation S240 transmits the extraction result to a repository for the terminal data application system to extract, including operation S241.

In operation S241, the extraction result is queued in a multi-threaded manner to a repository for the terminal data application system to extract. Thereby, the operation S241 can facilitate the transmission of the extraction result to the repository for the terminal data application system to extract. Meanwhile, the extraction result is transmitted in a multi-thread mode queue, so that the transmission speed can be increased, the terminal data application system can be responded quickly, and the support of the terminal data application system is improved.

According to some embodiments of the present disclosure, as shown in fig. 6, before the operation S240 transmits the extraction result to the repository for the terminal data application system to extract, the event stream processing method further includes an operation S002.

In operation S002, the extracted result is formatted and converted according to the data format required by the terminal data application system. Therefore, the extraction result can be conveniently and directly applied to the terminal data application system, the step of reconverting the extraction result by the terminal data application system is reduced, and the application convenience is improved.

Operation S240 transmits the extraction result to a repository for the terminal data application system to extract, including: and transmitting the extraction result after the formatting conversion to a storage library for the extraction of the terminal data application system. Therefore, the extraction result can better meet the requirements of the terminal data application system, and the terminal data application system can conveniently apply the data.

As some specific examples, the terminal data application system may be a bank credit card business system, and the bank credit card business system needs to perform security operations such as risk judgment and fraud prevention on the card swiping transaction of the user, so the bank credit card business system may put forward a need for extracting risk features from the transaction information, that is, the request information may be used for extracting risk features from the transaction information.

At this time, the data source system may be a transaction system, the customer generates original data after swiping a credit card, the original data may be transaction information, the original transaction information may include data such as a card number, user information, time, a consumption scene, a consumption type, a consumption amount, and the like, and the transaction information may be subjected to structure conversion and addition or deletion of data fields according to a pre-configured data structure. The resulting integrated data may be, for example, consumption type, consumption amount, and consumption scenario.

The integrated data of one time window of 1 second can be aggregated, namely, the consumption type, the consumption amount and the consumption scene are aggregated, the risk characteristics can be extracted from the aggregation result, and the risk characteristics are merged and classified into the current window event of 1 s. The window event can be extracted by a set time period. And converting the extracted extraction result into structured data. For example, the structured extraction results can be shown in table 1.

TABLE 1

Serial number

Card number

Type of consumption

Amount of consumption

Consumption scenario

Risk characterization

1

23344

Living goods

5000

On-line transactions

Small

2

15644

Pipe product

8000

Overseas

In

Therefore, the extraction result can be transmitted to the storage library for the extraction of the bank credit card business system.

As some specific examples, the terminal data application system may be a log early warning analysis system, the data source system may be another service system, the raw data may be log information generated by another service system, and the request information provided by the log early warning analysis system may be log information at the beginning of Error in the raw data recorded as Error log information output by another service system, and the Error log information is counted, merged and classified.

The generated log information can be collected in real time through an automatic script and transmitted into a data stream pipeline. And carrying out format conversion and data cleaning on the real-time log information. The log information of a time window (e.g. 1 minute) is configured with key fields according to a pre-configured data structure, such as a system code number, start information, log content, and the like. And further, the method can record the log information at the head of Error as Error reporting log information output by other service systems, and carry out quantity statistics and specific Error reporting information merging and classification on 1min Error logs from a time window. If the error data of 1h needs to be counted, only the window events of the previous 60 time windows of 1min need to be extracted and provided for the subsequent operation. And converting the extracted extraction result into structured data. For example, the structured extraction results can be shown in tables 2 and 3.

TABLE 2

Serial number	System code	Number of error reports
			1	A	200
2	B	20

TABLE 3

Serial number	System code number	Log content
			1	A	XXXX.XX.XX XX.XX.XX
2	B	XXXX.XX.XX XX.XX.XX

Therefore, the extraction result can be transmitted to a storage library for being extracted by a log early warning analysis system.

Based on the event stream processing method, the present disclosure also provides an event stream processing apparatus 10. The event stream processing apparatus 10 will be described in detail below with reference to fig. 7.

Fig. 7 schematically shows a block diagram of the event stream processing apparatus 10 according to the embodiment of the present disclosure.

The event stream processing device 10 comprises a collecting device 1, an aggregation device 2, an extraction device 3 and a transmission device 4.

Acquisition apparatus 1, acquisition apparatus 1 being configured to perform operation S210: collecting raw data from m data source systems, wherein the raw data has a collection time stamp, and m is an integer greater than or equal to 1.

The aggregation device 2, the aggregation device 2 is configured to perform operation S220: and aggregating the original data in a preset time window according to the acquisition timestamp based on the request information of the terminal data application system to obtain a window event, wherein the request information is a specific requirement for aggregating the original data.

Extracting means 3, the extracting means 3 being configured to perform operation S230: and extracting the window event according to the set time period to obtain an extraction result.

A transmission device 4, the transmission device 4 being configured to perform operation S240: and transmitting the extraction result to a storage library for the terminal data application system to extract.

According to the event stream processing apparatus 10 of the embodiment of the present disclosure, based on the request information of the terminal data application system, the original data in the preset time window is aggregated according to the acquisition timestamp to obtain the window event, and the window event is extracted according to the set time period to obtain the extraction result. The use requirement of the terminal data application system can be met only by transmitting the extraction result to the storage library, so that the original data before aggregation does not need to be transmitted, and the consumption of network resources and memory resources is directly reduced; in addition, because the target expressions of the window events in each time window are the same, the window events can be merged across the time windows under the condition of not amplifying errors to obtain an extraction result, and a summarizing effect is obtained under the condition of not losing accuracy, so that the memory amount required by calculation per second can be reduced, the internal memory consumption can be reduced to the maximum extent, the increment calculation amount is reduced, the calculation resources are saved, and the relative accuracy of the aggregation module with the aggregation function is further maintained.

In addition, according to the embodiment of the present disclosure, any plurality of modules in the collecting device 1, the aggregating device 2, the extracting device 3, and the transmitting device 4 may be combined into one module to be implemented, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.

According to an embodiment of the present disclosure, at least one of the collecting device 1, the aggregating device 2, the extracting device 3 and the transmitting device 4 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or may be implemented by any one of three implementations of software, hardware and firmware, or any suitable combination of any of them.

Alternatively, at least one of the collecting means 1, the aggregating means 2, the extracting means 3 and the transmitting means 4 may be at least partly implemented as a computer program module, which when executed may perform a corresponding function.

Fig. 8 schematically shows a block diagram of an electronic device adapted to implement the above method according to an embodiment of the present disclosure.

As shown in fig. 8, an electronic apparatus 900 according to an embodiment of the present disclosure includes a processor 901 which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. Processor 901 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or related chipset(s) and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), and/or the like. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 are stored. The processor 901, ROM 902, and RAM 903 are connected to each other by a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 900 may also include input/output (I/O) interface 905, input/output (I/O) interface 905 also connected to bus 904, according to an embodiment of the present disclosure. The electronic device 900 may also include one or more of the following components connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The driver 910 is also connected to an input/output (I/O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or RAM 903 described above and/or one or more memories other than the ROM 902 and RAM 903.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. The program code is for causing a computer system to perform the methods of the embodiments of the disclosure when the computer program product is run on the computer system.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 901. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, and the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, and downloaded and installed through the communication section 909 and/or installed from the removable medium 911. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the disclosure, and these alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. An event stream processing method, comprising:

collecting original data from m data source systems, wherein the original data has a collection time stamp, and m is an integer greater than or equal to 1;

aggregating the original data in a preset time window according to the acquisition timestamp based on request information of a terminal data application system to obtain a window event, wherein the request information is a specific requirement for aggregating the original data;

extracting the window event according to a set time period to obtain an extraction result; and

and transmitting the extraction result to a storage library for the terminal data application system to extract.

2. The event stream processing method according to claim 1, wherein the collecting raw data from each data source system comprises:

setting a data interface device for each data source system, formulating m external protocols, wherein the m external protocols correspond to the m data source systems one by one, and acquiring different types of data generated by the m data source systems to a data flow pipeline through the m external protocols, wherein the event flow processing method is realized in the data flow pipeline.

3. The event stream processing method according to claim 1, wherein before the request information based on the terminal data application system aggregates the original data within a preset time window according to the collection timestamp to obtain a window event, the event stream processing method further comprises:

integrating the original data according to a pre-configured data structure to obtain integrated data, wherein the pre-configured data structure comprises a data format and/or a data field;

the acquiring method comprises the following steps that the request information based on the terminal data application system aggregates the original data in a preset time window according to the acquisition time stamp to obtain a window event, and comprises the following steps:

and aggregating the integrated data in a preset time window according to the acquisition time stamp based on the request information of the terminal data application system to obtain a window event.

4. The event stream processing method according to claim 3, wherein the aggregating the integrated data in a preset time window according to the collection timestamp based on the request information of the terminal data application system to obtain a window event comprises:

and calculating the integration data in a preset time window by using a polymerization algorithm according to the acquisition time stamp based on the request information of the terminal data application system to obtain a window event.

5. The event stream processing method according to claim 1, wherein the transmitting the extraction result to a repository for the terminal data application system to extract comprises:

and queuing and transmitting the extraction results to a storage library in a multithreading mode for the terminal data application system to extract.

6. The event stream processing method according to any one of claims 1 to 5, wherein before the transmitting the extraction result to a repository for extraction by the terminal data application system, the event stream processing method further comprises:

carrying out formatting conversion on the extraction result according to a data format required by the terminal data application system;

the transmitting the extraction result to a repository for the terminal data application system to extract includes:

and transmitting the extraction result after the formatting conversion to a storage library for the extraction of the terminal data application system.

7. An event stream processing apparatus, comprising:

the device comprises a collecting device and a processing device, wherein the collecting device is used for collecting original data from m data source systems, the original data has a collecting time stamp, and m is an integer greater than or equal to 1;

the aggregation device is used for executing request information based on a terminal data application system, aggregating the original data in a preset time window according to the acquisition timestamp to obtain a window event, wherein the request information is a specific requirement for aggregating the original data;

the extraction device is used for extracting the window event according to a set time period to obtain an extraction result; and

and the transmission device is used for transmitting the extraction result to a storage library for the extraction of the terminal data application system.

8. An electronic device, comprising:

one or more processors;

one or more memories for storing executable instructions that, when executed by the processor, implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that the storage medium has stored thereon executable instructions which, when executed by a processor, implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program comprising one or more executable instructions which, when executed by a processor, implement the method according to any one of claims 1 to 6.