CN115062002A - Stream data processing method and device - Google Patents

Abstract

The embodiment of the invention provides a streaming data processing method and device. Wherein, the method comprises the following steps: determining corresponding configuration parameters according to the current target log; in response to receiving at least one rule selected by a user, determining a target model according to the at least one rule and configuration parameters, wherein the data processing capacity of the target model is matched with the target log; processing the target log through a correlation analysis engine according to the target model to obtain and store a corresponding flow analysis result; and reading the flow analysis result of the preset time period, carrying out global weight judgment and global statistics on the flow analysis result of the preset time period through the bloom filter to obtain a global flow analysis result, and storing the global flow analysis result in a database. The method and the device realize the judgment, merging, counting, whitening and grouping of real-time streaming data.

Description

Stream data processing method and device

technical field

The present invention relates to the field of computer technology, and in particular, to a method and device for processing stream data.

Background technique

Bloom filter is essentially a relatively clever probabilistic data structure (binary vector), which stores either 0 or 1.

At present, most of the bloom filter techniques are used for problems such as deduplication based on a limited amount of data, for example: Google's distributed database Bigtable uses bloom filters to find non-existent rows or columns; Google Chrome uses Bloom filters Bloom filters speed up safe browsing services; the SPIN model detector uses bloom filters to track reachable state spaces when validating problems at scale; the Venti document storage system also employs bloom filters to detect previously stored data. This will inevitably lead to problems such as limited storage data volume and insufficient scalability of the Bloom filter.

SUMMARY OF THE INVENTION

In view of the problems in the prior art, embodiments of the present invention provide a method and apparatus for processing stream data.

Specifically, the embodiments of the present invention provide the following technical solutions:

In a first aspect, an embodiment of the present invention provides a streaming data processing method, including: determining a corresponding configuration parameter according to a current target log; in response to receiving at least one rule selected by a user, according to the at least one rule and all The configuration parameters determine the target model, and the data processing capability of the target model matches the size of the target log; according to the target model, the target log is processed by the correlation analysis engine, and the corresponding traffic analysis result is obtained and stored. ; Read the traffic analysis results of the preset time period, perform global judgment and global statistics on the traffic analysis results of the preset time period through the Bloom filter, obtain the global traffic analysis results, and use the global traffic analysis results. saved in the database.

Further, the association analysis engine includes a Sabre engine.

Further, the configuration parameters include at least one of the following: a traffic size corresponding to the target log, a blacklist corresponding to the target log, a storage address corresponding to the target log, a target field corresponding to the target log, and a target field corresponding to the target log. The merge field for the target log.

Further, before determining the corresponding configuration parameter according to the current target log, the method further includes: presetting at least one initial rule, and the at least one initial rule is used to be selected by the user.

Further, read the traffic analysis results of the preset time period, perform global weighting and global statistics on the traffic analysis results of the preset time period through the Bloom filter, obtain the global traffic analysis results, and use the global traffic analysis results. The analysis results are stored in the database, including: reading the traffic analysis results of a preset time period, and performing global weighting and global statistics on the traffic analysis results of the preset time period through a Bloom filter, so as to obtain the global traffic analysis results; The global flow analysis result is converted into a corresponding binary vector and stored in the Bloom filter; the global flow analysis result is stored in a database.

Further, the method further includes: setting a scheduled deletion task, clearing the data in the database according to the scheduled deletion task, and setting a binary vector corresponding to the data in the Bloom filter to zero.

Further, according to the target model, the target log is processed by an association analysis engine, including: according to the target model, the target log is judged, merged, counted, and whitened by an association analysis engine. and grouping.

In a second aspect, an embodiment of the present invention further provides a streaming data processing apparatus, including: a first processing module, configured to determine a corresponding configuration parameter according to a current target log; a second processing module, configured to respond to the received At least one rule selected by the user, a target model is determined according to the at least one rule and the configuration parameter, and the data processing capability of the target model matches the size of the target log; the third processing module is used for according to the target model. The model, processes the target log through the correlation analysis engine, and obtains and stores the corresponding traffic analysis result; the fourth processing module is used for reading the traffic analysis result of the preset time period, It is assumed that the traffic analysis results of the time period are subjected to global weight judgment and global statistics to obtain the global traffic analysis results, and the global traffic analysis results are stored in the database.

In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the first program when executing the program The steps of the streaming data processing method described in the aspect.

In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the stream data processing method described in the first aspect. step.

In a fifth aspect, an embodiment of the present invention further provides a computer program product that stores executable instructions thereon, and when the instructions are executed by a processor, enables the processor to implement the steps of the streaming data processing method described in the first aspect.

The streaming data processing method and device provided by the embodiments of the present invention perform real-time calculation and real-time statistics on streaming data through an association analysis engine according to different target models constructed, so as to realize the screening and filtering of data sets. Then, the current streaming data is processed through the bloom filter, which expands the data processing capability of the bloom filter; the data can be efficiently marked by the bloom filter to realize the global statistics and global judgment of the streaming data of the preset time period. Heavy; the Bloom filter only performs global weighting and global statistics on the traffic analysis results to obtain the global traffic analysis results, and saves the data corresponding to the global traffic analysis results in the database, which solves the problem of limited data storage by the Bloom filter.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flow chart of an embodiment of a streaming data processing method according to the present invention;

Fig. 2 is the frame schematic diagram of the stream data processing method;

Fig. 3 is the design schematic diagram of the business module of the stream data processing method;

4 is a schematic structural diagram of an embodiment of a streaming data processing apparatus according to the present invention;

FIG. 5 is a schematic structural diagram of a physical embodiment of an electronic device according to the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

FIG. 1 is a flowchart of an embodiment of a stream data processing method according to the present invention. As shown in FIG. 1 , the streaming data processing method according to the embodiment of the present invention includes:

S101: Determine corresponding configuration parameters according to the current target log.

The target log is the traffic log, that is, the log that records the traffic, including the size of the inbound and outbound traffic, click stream data, etc. The larger the traffic data, the larger the corresponding traffic log.

As an example, traffic logs may include timestamps, source IP, destination IP, source port, destination port, incoming and outgoing traffic, etc. Typically, a stream is aggregated into one record and sent to the log server. A flow refers to the same source IP, destination IP and destination port.

As an example, the current target log may be the traffic log of the previous second.

As an example, if the traffic log only includes the timestamp and the size of incoming and outgoing traffic, and the current target log is the traffic log of the previous second, then the configuration parameters corresponding to the current target log can be obtained from the traffic log: target field, time, incoming and outgoing traffic size.

S102, in response to receiving at least one rule selected by the user, determine a target model according to the at least one rule and configuration parameters, where the data processing capability of the target model matches the size of the target log.

The configuration parameters are dynamically adjusted based on changes in traffic, and at least one rule is based on a combination of different rules selected by the user, so that the target model is determined according to the two dynamic factors of the at least one rule and the configuration parameter.

As an example, the human-computer interaction interface may display the names of multiple scenarios, and the user may select at least one rule by selecting at least one scenario through the human-computer interaction interface. The human-computer interaction interface can also display the names of multiple preset rules and the uses of the rules (for example, a rule is used for weighting, merging, counting, whitening or grouping) for user selection.

As an example, the user can select scenario 1 (rule 1 corresponding to scenario 1 is used to judge traffic data) and scenario 2 (rule 2 corresponding to scenario 2 is used to merge traffic data) through the human-computer interaction interface, and the configuration parameters can be The size of the incoming and outgoing traffic obtained from the traffic log (for example, 10MB), then the target model determined according to rule 1 and rule 2 and the configuration parameters can be a model that can judge and merge data with a traffic size of 10MB.

As an example, the target model may be determined according to at least one rule selected by the user, configuration parameters, and authentication information. Authentication information is used to perform security authentication during data interaction.

The target model is determined according to at least one rule and configuration parameters, so as to encapsulate multiple rules to build different models, filter out multiple data structures, and adapt to different scenarios.

S103, according to the target model, the target log is processed by the correlation analysis engine, and the corresponding traffic analysis result is obtained and stored.

The correlation analysis engine (such as the Sabre engine) is a big data streaming distributed correlation analysis engine that can realize real-time calculation and real-time statistics.

As an example, according to the current target log, the corresponding target model is determined, and then, the correlation analysis engine performs weighting, merging, counting, whitening and grouping on the current target log according to the target model, and completes the real-time analysis of the pair. Store the traffic analysis results of real-time analysis. This application does not limit the storage method.

The correlation analysis engine may be a streaming correlation analysis engine capable of real-time calculation and real-time statistics of big data.

S104: Read the traffic analysis result of the preset time period, perform global weighting and global statistics on the traffic analysis result of the preset time period through the Bloom filter, obtain the global traffic analysis result, and save the global traffic analysis result in the database .

In some instances, the above traffic analysis results are stored in real time, for example, the traffic analysis results are stored in real time every second, and when a certain amount of traffic analysis results are stored, the traffic analysis results in a preset time period can be read. As an example, if the traffic analysis results for the first two hours have been stored and the preset time period is two hours, read the traffic analysis results of the two hours, and use the Bloom filter to analyze the traffic analysis results in the preset time period. Carry out global weight judgment and global statistics (that is, obtain the judgment weight and statistical results of the first two hours), obtain the global traffic analysis results, and save the global traffic analysis results in the database. The database type and the storage manner of the global traffic analysis result are not limited in the present invention.

Bloom filter, which is essentially a binary data structure, is used to determine whether an element (key) is in a certain set. The Bloom algorithm used in the Bloom filter is a weighting algorithm based on binary data sets. Through the Bloom filter, the global judgment and global statistics of streaming data are realized.

The streaming data processing method and device provided by the embodiments of the present invention perform real-time calculation and real-time statistics on streaming data through an association analysis engine according to different target models constructed, so as to realize the screening and filtering of data sets. Then, the current streaming data is processed through the bloom filter, which expands the data processing capability of the bloom filter; the data can be efficiently marked by the bloom filter to realize the global statistics and global judgment of the streaming data of the preset time period. Heavy; the Bloom filter only performs global weighting and global statistics on the traffic analysis results to obtain the global traffic analysis results, and saves the data corresponding to the global traffic analysis results in the database, which solves the problem of limited data storage by the Bloom filter.

On the basis of the above embodiment, the association analysis engine may be a Sabre engine.

Sabre engine is a stream correlation analysis engine that can calculate and count big data in real time.

The streaming data processing method provided by the embodiment of the present invention adopts the advantages of the real-time calculation and real-time statistics of big data of the Sabre engine, which further facilitates the real-time calculation and real-time statistics of the current target log.

On the basis of the above embodiment, the configuration parameters may include at least one of the following: the traffic size corresponding to the target log, the blacklist corresponding to the target log, the storage address of the corresponding target log, the target field corresponding to the target log, and the merging of the corresponding target log field.

Configuration parameters can be obtained from the current target log. For example, the traffic size, storage address, and target fields can be directly read in the corresponding target log, and the blacklist can be determined after analyzing the target fields. Different types of target logs correspond to different blacklists. Get information based on the target field in the target log. Integrate the acquired information according to the merge fields. Similarly, configuration parameters can also include a whitelist of corresponding target logs. The configuration parameters of cyclic regulation data processing are realized, and the target model of different data volume is adapted.

On the basis of the above embodiment, before determining the corresponding configuration parameter according to the current target log, the method may further include: preset at least one initial rule, and at least one initial rule is used for selection by the user.

The preset at least one initial rule may be weighting, merging, counting, whitening or grouping for different types of target logs. Based on the different initial rules selected by the user, new rules can be combined to continuously expand the optional rules dynamically.

On the basis of the above-mentioned embodiment, the traffic analysis results of the preset time period are read, and the traffic analysis results of the preset time period are globally judged and counted through the Bloom filter to obtain the global traffic analysis results, and the global traffic analysis results are obtained. The traffic analysis results are stored in the database, which may include: reading the traffic analysis results of the preset time period, and performing global weighting and global statistics on the traffic analysis results of the preset time period through the Bloom filter, so as to obtain the global traffic analysis results; The global traffic analysis results are converted into corresponding binary vectors and stored in the Bloom filter; the global traffic analysis results are stored in the database.

The binary vector saved in the bloom filter corresponds to the global traffic analysis result saved in the database.

Based on the above embodiment, the method may further include: setting a scheduled deletion task, cleaning data in the database according to the scheduled deletion task, and setting the binary vector of the corresponding data in the Bloom filter to zero.

The scheduled deletion task may also be a deletion task set according to time. For example, set the Bloom filter and the database to store only one week's data. If the next week's data needs to be stored, the data with the earliest storage time needs to be deleted.

The scheduled deletion task can also set the maximum number of stored data in the bloom filter and database. If the bloom filter and database reach the storage limit, when new data needs to be stored, the one with the longest storage time will be deleted accordingly. data.

The delete (reset) function of the bloom filter is implemented by zeroing the binary vector of the corresponding data in the bloom filter.

On the basis of the above embodiment, according to the target model, the target log is processed by the correlation analysis engine, including: according to the target model, the target log is judged, merged, and counted by the correlation analysis engine. , whitening and grouping.

To judge the severity, it may be to judge whether the data in the log is duplicated. As an example, compare the contents of the target fields in the log to eliminate duplicate contents.

Merge, which can be the integration of eligible data in the log.

The count can be to calculate how many data segments exist in the log, or to calculate the sum of duplicate data in the log, etc. You can count data for a specified feature as needed.

Adding white can be to directly filter out the data in the whitelist according to a preset whitelist, or to keep the data in the blacklist according to a preset blacklist. The log traffic can also be screened according to the whitelist and the blacklist at the same time, which is not limited in the present invention. The contents of the whitelist and blacklist can be set according to specific needs.

The grouping can be to group the log traffic data according to the target field in the log. For example, the target field is id, and the field of log traffic data includes id and the data content of the corresponding id. If the target field is all even-numbered ids, then all even-numbered fields can be grouped into one group, and all remaining odd-numbered fields can be grouped into one group.

As shown in FIG. 2 , according to the framework diagram of the solution of the above embodiment, the functions of filtering, global weighting and global statistics of streaming (traffic) data are realized by using Bloom filter combined with a streaming analysis engine (sabre engine). The system initializes a number of preset (configurable) rules, which are used to form data sources for other rules. Configuration parameters are automatically adjusted depending on changes in the amount of streaming data, and are regularly synchronized to the model. The model is updated with changes in rules and configurations. In response to changes in the amount of streaming data, the model will be automatically adjusted, and the corresponding data processing capabilities and storage capabilities will also be adjusted accordingly, until the amount of data accessed and the processing capability are balanced.

Reference may be made to FIG. 3 for the design of the service module of the solution according to the above embodiment. The content of Request in FIG. 3 may be: subject attribute (eg, request format), object resource attribute (eg, the id of the selected rule), rule name, and start/stop information of the rule. The content of Response can be: Allow, Deny and Error.

1. The system presets the rules supported by default, and the front-end can directly select different scenarios to combine;

2. The user selects different scenarios to combine different rules, and the sabre engine receives these rules and builds different data processing models;

3. The engine processes data (weighting, merging, counting, whitening and grouping) according to the built model, and different models output different data sets;

4. Calculate the output data into a corresponding binary vector according to the Bloom algorithm, and save it to the Bloom filter;

5. The warehousing object saves the filtered and weighted data to the database for persistence and records it synchronously in the Bloom filter for marking;

6. The business module queries the database to obtain the latest dataset;

7. The scheduled task object synchronously cleans the database and deletes the data in the Bloom filter (zero) according to the amount of data stored in the database and storage time;

8. Monitor the changes in the amount of streaming data, and adjust the corresponding configuration parameters adaptively. The sabre engine applies the latest models and configuration parameters to dynamically adjust the output data of data processing to achieve a balance between the amount of streaming data and data processing capabilities.

In summary, based on the preset rules, a variety of data processing models are constructed, providing a more efficient and convenient means to filter out the data needed in streaming data and adapt to different scenarios. In response to changes in the amount of streaming data, the model is automatically adjusted to achieve the effect of fast processing of streaming data.

FIG. 4 is a schematic structural diagram of an embodiment of a streaming data processing apparatus according to the present invention. As shown in Figure 4, the stream data processing device includes:

The first processing module 401 is configured to determine corresponding configuration parameters according to the current target log;

The second processing module 402 is configured to, in response to receiving at least one rule selected by the user, determine a target model according to at least one rule and configuration parameters, and the data processing capability of the target model matches the size of the target log;

The third processing module 403 is configured to process the target log through the correlation analysis engine according to the target model, and obtain and store the corresponding traffic analysis result;

The fourth processing module 404 is used to read the traffic analysis results of the preset time period, and perform global weighting and global statistics on the traffic analysis results of the preset time period through the Bloom filter, so as to obtain the global traffic analysis results, and analyze the global traffic analysis results. The traffic analysis results are saved in the database.

Optionally, the association analysis engine includes a Sabre engine.

Optionally, the configuration parameters include at least one of the following: a traffic size corresponding to the target log, a blacklist corresponding to the target log, a storage address corresponding to the target log, a target field corresponding to the target log, and a merge field corresponding to the target log.

Optionally, the device further includes:

The fifth processing module 405 is configured to preset at least one initial rule, and the at least one initial rule is used to be selected by the user.

Optionally, the fourth processing module 404 is used for:

Read the traffic analysis results of the preset time period, and perform global weighting and global statistics on the traffic analysis results of the preset time period through the Bloom filter to obtain the global traffic analysis results;

Convert the global traffic analysis result to the corresponding binary vector and save it in the Bloom filter;

Save the global traffic analysis results in the database.

Optionally, the device further includes:

The sixth processing module 406 is configured to set the scheduled deletion task, clean up the data in the database according to the scheduled deletion task, and set the binary vector of the corresponding data in the Bloom filter to zero.

Optionally, the third processing module 403 is further configured to: according to the target model, perform weighting, merging, counting, whitening, and grouping on the target logs through an association analysis engine.

An example is as follows:

FIG. 5 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 5 , the electronic device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein , the processor 501 , the communication interface 502 , and the memory 503 communicate with each other through the communication bus 504 . The processor 501 can invoke the logic instructions in the memory 503 to perform the following method: determine the corresponding configuration parameter according to the current target log; in response to receiving at least one rule selected by the user, determine the target model according to the at least one rule and the configuration parameter, The data processing capability of the target model matches the size of the target log; according to the target model, the target log is processed by the correlation analysis engine to obtain and store the corresponding traffic analysis results; The filter performs global weighting and global statistics on the traffic analysis results of the preset time period, obtains the global traffic analysis results, and saves the global traffic analysis results in the database.

In addition, the above-mentioned logic instructions in the memory 503 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer program The streaming data processing methods provided by the above embodiments can be executed, for example, including: determining the corresponding configuration parameters according to the current target log; in response to receiving at least one rule selected by the user, determining the target model according to the at least one rule and the configuration parameters, The data processing capability of the target model matches the size of the target log; according to the target model, the target log is processed by the correlation analysis engine to obtain and store the corresponding traffic analysis results; The filter performs global weighting and global statistics on the traffic analysis results of the preset time period, obtains the global traffic analysis results, and saves the global traffic analysis results in the database.

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and the computer program is implemented when executed by a processor to execute the stream data processing methods provided by the above embodiments, for example Including: determining the corresponding configuration parameter according to the current target log; in response to receiving at least one rule selected by the user, determining the target model according to the at least one rule and the configuration parameter, and the data processing capability of the target model matches the size of the target log; according to the target The model processes the target log through the correlation analysis engine, and obtains and stores the corresponding traffic analysis results; reads the traffic analysis results of the preset time period, and performs global judgment on the traffic analysis results of the preset time period through the Bloom filter. and global statistics, obtain the global traffic analysis results, and save the global traffic analysis results in the database.

The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place , or distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic Disks, optical discs, etc., include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods of various embodiments or portions of embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (11)

1. A method of streaming data processing, the method comprising:

determining corresponding configuration parameters according to the current target log;

in response to receiving at least one rule selected by a user, determining a target model according to the at least one rule and the configuration parameters, wherein the data processing capacity of the target model is matched with the size of the target log;

processing the target log through a correlation analysis engine according to the target model to obtain and store a corresponding flow analysis result;

reading a flow analysis result of a preset time period, carrying out global duplication judgment and global statistics on the flow analysis result of the preset time period through a bloom filter to obtain a global flow analysis result, and storing the global flow analysis result in a database.

2. The streaming data processing method of claim 1, wherein the correlation analysis engine comprises a Sabre engine.

3. The streaming data processing method according to any of claims 1 to 2, wherein the configuration parameters include at least one of: the flow corresponding to the target log, the blacklist corresponding to the target log, the storage address corresponding to the target log, the target field corresponding to the target log and the merging field corresponding to the target log.

4. The streaming data processing method according to any one of claims 1 to 2, wherein before determining the corresponding configuration parameter according to the current target log, the method further comprises:

at least one initial rule is preset, the at least one initial rule being for selection by a user.

5. The streaming data processing method according to any one of claims 1 to 2, wherein the reading of the traffic analysis result in the preset time period, performing global re-judgment and global statistics on the traffic analysis result in the preset time period through a bloom filter to obtain a global traffic analysis result, and storing the global traffic analysis result in a database includes:

reading a flow analysis result of a preset time period, and performing global weight judgment and global statistics on the flow analysis result of the preset time period through a bloom filter to obtain a global flow analysis result;

converting the global flow analysis result into a corresponding binary vector and storing the binary vector in the bloom filter;

and storing the global flow analysis result in a database.

6. The streaming data processing method of claim 5, wherein the method further comprises:

and setting a timed deleting task, clearing the data in the database according to the timed deleting task, and setting the binary vector corresponding to the data in the bloom filter to zero.

7. The streaming data processing method of claim 1, wherein the processing the target log by a correlation analysis engine according to the target model comprises:

and judging, merging, counting, whitening and grouping the target logs through a correlation analysis engine according to the target model.

8. A streaming data processing apparatus, characterized in that the method comprises:

the first processing module is used for determining corresponding configuration parameters according to the current target log;

the second processing module is used for responding to at least one rule selected by a user, determining a target model according to the at least one rule and the configuration parameters, and the data processing capacity of the target model is matched with the size of the target log;

the third processing module is used for processing the target log through a correlation analysis engine according to the target model to obtain and store a corresponding flow analysis result;

and the fourth processing module is used for reading the flow analysis result of the preset time period, performing global duplication judgment and global statistics on the flow analysis result of the preset time period through the bloom filter to obtain a global flow analysis result, and storing the global flow analysis result in the database.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the streaming data processing method according to any of claims 1 to 6 are implemented when the processor executes the program.

10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the streaming data processing method according to any one of claims 1 to 6.

11. A computer program product having stored thereon executable instructions, characterized in that the instructions, when executed by a processor, cause the processor to carry out the steps of the streaming data processing method according to any of claims 1 to 6.

Images