WO2017008650A1 - 一种用于过滤数据的设备及方法 - Google Patents

一种用于过滤数据的设备及方法 Download PDF

Info

Publication number
WO2017008650A1
WO2017008650A1 PCT/CN2016/088302 CN2016088302W WO2017008650A1 WO 2017008650 A1 WO2017008650 A1 WO 2017008650A1 CN 2016088302 W CN2016088302 W CN 2016088302W WO 2017008650 A1 WO2017008650 A1 WO 2017008650A1
Authority
WO
WIPO (PCT)
Prior art keywords
rule
data
filtering
filtered
abstract syntax
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/088302
Other languages
English (en)
French (fr)
Inventor
丁崔灿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017008650A1 publication Critical patent/WO2017008650A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2457—Query processing with adaptation to user needs
    • G06F16/24578—Query processing with adaptation to user needs using ranking

Definitions

  • the present application relates to the field of computers, and in particular, to a technique for filtering data satisfying a filtering rule in real time from massive data according to a set filtering rule.
  • the SQL statement based on the memory relational database is used to filter the valid data.
  • this method needs to cache the massive data in the logical data table of the in-memory database, which occupies a large amount of memory resources, and the periodic execution of the SQL statement is difficult to achieve. Real-time requirements;
  • a massive data storage scheme based on Hbase (a distributed, column-oriented open source database) that uses Map-Reduce algorithm (a programming model algorithm for parallel operations on large-scale data sets) to filter valid data, however, Map The -Reduce model task is similar to the post-calculation mode of batch processing.
  • Map-Reduce algorithm a programming model algorithm for parallel operations on large-scale data sets
  • Map The -Reduce model task is similar to the post-calculation mode of batch processing.
  • For massive data already stored in Hbase only the operation matching result can be executed periodically, real-time performance is difficult to be guaranteed, and complex Map-Reduce model tasks need to pass. Extended writing to achieve, it is difficult to meet the needs of real-time variable and multiple calculations for a large number of filtering rules;
  • CEP engine Complex Event Processing
  • using pattern matching algorithm to filter valid data is more suitable for enterprise application system monitoring and decision control.
  • mature CEP engine is mostly commercial software, user cost is high, and CEP engine Each has its own pattern rule description method.
  • Drools uses XML format
  • Esper uses EPL format, and needs to write a large number of adaptation codes for different system requirements.
  • the difficulty of implementation in addition, the implementation of the CEP engine is different, so it is not easy to monitor and tune the performance of the CEP engine.
  • the technical problem to be solved by the present application is how to filter out the filtering rules in real time from the massive data according to the set filtering rules without occupying a large amount of memory resources, and can satisfy a large number of filtering rules in real time and various kinds. Calculated demand.
  • the present application provides a method for filtering data, wherein the method includes:
  • each of the filtering rules includes a rule domain identifier, a rule name, and a rule operation expression, and a first rule list of the filtering rule indexed by the domain identifier of the filtering rule is established;
  • the parallelized matching operation is performed on the structured to-be-filtered data by using the obtained plurality of filtering rules.
  • the obtaining the initial data to be filtered includes:
  • the initial to-be-filtered data is obtained from the distributed message middleware.
  • converting the initial data to be filtered into structured data to be filtered further includes:
  • Obtaining the structured data to be filtered includes:
  • performing parallel matching operations on the structured to-be-filtered data by using the obtained plurality of filtering rules includes:
  • performing rule compilation on the obtained filtering rule to establish a runnable abstract syntax tree includes:
  • performing a pre-calculation on the abstract syntax tree includes:
  • the two operands corresponding to the operator are transmitted out of the running stack, and the calculation is performed to obtain a calculation result;
  • the special element is converted into a programming language data structure element and passed to the running stack.
  • parallel matching calculations using a plurality of said runnable abstract syntax trees include:
  • a matching calculation is performed on the runnable abstract syntax tree using the run stack.
  • the method further includes:
  • the first rule list of the filtering rule that is set to be indexed by the domain identifier of the filtering rule further includes:
  • the newly added filtering rule, deleting the filtering rule, or modifying the existing filtering rule to compile includes at least one of the following:
  • each of the filtering rules further includes: information about a notifier to which the filtering rule is bound;
  • the method further includes:
  • the structured to-be-filtered data that satisfies the corresponding filtering rule is sent to the notifier bound to the filtering rule for transmission.
  • an apparatus for filtering data comprising:
  • a first device configured to acquire initial data to be filtered, and convert the initial data to be filtered into structured data to be filtered, where the structured data to be filtered includes a data domain identifier and a data body of a key-value pair format ;
  • a second device configured to load a filtering rule, where each of the filtering rules includes a rule domain identifier, a rule name, and a rule operation expression, and establishes a filtering rule that is indexed by a domain identifier of the filtering rule. a list of rules;
  • a third device configured to acquire the structured to-be-filtered data, and obtain, according to the data domain identifier, a plurality of filtering rules that have a rule domain identifier corresponding to the data domain identifier from the first rule list;
  • a fourth device configured to perform parallel matching on the structured data to be filtered by using the obtained plurality of filtering rules Operation.
  • the first device includes:
  • the unit of the initial data to be filtered is obtained from the distributed message middleware.
  • the first device includes:
  • the third device includes:
  • the fourth device includes:
  • the unit for performing rule compilation on the acquired filtering rule to establish a runable abstract syntax tree includes:
  • the two operands corresponding to the operator are transmitted out of the running stack, and the calculation is performed to obtain a calculation result.
  • the special element is converted into a program language data structure element and then passed into the running stack.
  • the unit for using the data body of the structured data to be filtered as an input parameter, traversing a plurality of the runnable abstract syntax trees, and performing parallel matching calculation by using the plurality of the runnable abstract syntax trees includes:
  • a module for performing a matching calculation on the runnable abstract syntax tree using the run stack is a module for performing a matching calculation on the runnable abstract syntax tree using the run stack.
  • the device further includes:
  • the fifth device is configured to add a filtering rule, delete a filtering rule, or modify and compile an existing filtering rule.
  • the second device further includes:
  • the fifth device includes:
  • a unit for searching for a filter rule from the second rule list and modifying and compiling the searched filter rule is a unit for searching for a filter rule from the second rule list and modifying and compiling the searched filter rule.
  • each of the filtering rules further includes: information about a notifier to which the filtering rule is bound;
  • the device further includes:
  • a sixth device configured to send the structured to-be-filtered data that meets the corresponding filtering rule to the notifier bound to the filtering rule, for transmission.
  • the device and method for data filtering adopts a streaming operation mode, which does not cache or not solidify data in the memory, that is, each time the initial data to be filtered is acquired.
  • the initial data to be filtered is converted into structured data to be filtered, and the matching calculation is performed in real time by using the corresponding filtering rule, and then the filtering result is obtained, and the real-time problem of filtering the massive streaming data is solved;
  • the method and device for filtering data according to an apparatus and method for data filtering support arithmetic operations, string operations, relational operations, logical operations, regular expression operations, set operations,
  • the extended interface is reserved, and the filtering rule is a simple operation expression with a variable, which solves the problem that the filtering rule description is complicated, the expansion is difficult, and the management is difficult;
  • the device and method for data filtering provided by the present application according to an embodiment of the present application are independently designed and developed, the cost is relatively low, and can be monitored and tuned on any code path.
  • FIG. 1 shows a schematic diagram of an apparatus for filtering data according to an aspect of the present application
  • FIG. 2 is a schematic diagram of an apparatus for filtering data according to a preferred embodiment of the present application
  • FIG. 3 is a schematic diagram of an apparatus for filtering data according to another preferred embodiment of the present application.
  • FIG. 4 shows a flow chart of a method for filtering data provided in accordance with an aspect of the present application
  • FIG. 5 is a flowchart of a method for filtering data according to a preferred embodiment of the present application.
  • FIG. 6 is a flowchart of a method for filtering data according to another preferred embodiment of the present application.
  • FIG. 7 is a schematic diagram of an apparatus including the system for filtering data devices according to a preferred embodiment of the present application.
  • FIG. 8 to FIG. 10 are schematic diagrams showing parallel matching operations of the structured to-be-filtered data by using the acquired plurality of filtering rules according to a specific scenario of the present application.
  • the apparatus 1 shows a schematic diagram of an apparatus for filtering data according to an aspect of the present application, wherein the apparatus 1 includes: a first device 11, a second device 12, a third device 13, and a fourth device 14. .
  • the first device 11 is configured to obtain initial data to be filtered, and convert the initial data to be filtered into structured data to be filtered, where the structured data to be filtered includes a data domain identifier and a key-value.
  • a data body of the format the second device 12 is configured to load a filtering rule, where each of the filtering rules includes a rule domain identifier, a rule name, a rule operation expression, and establishes a domain identifier of the filtering rule as Indexing the first rule list of the filtering rule;
  • the third device 13 is configured to acquire the structured to-be-filtered data, and obtain the data from the first rule list according to the data domain identifier.
  • the domain identifier identifies a plurality of filtering rules of the rule domain identifiers; the fourth device 14 is configured to perform parallel matching operations on the structured data to be filtered by using the acquired plurality of filtering rules.
  • the first device 11 is configured to acquire initial data to be filtered, and convert the initial data to be filtered into structured data to be filtered, where the structured data to be filtered includes a data domain identifier and a key-value.
  • the structured data to be filtered includes a data body identifier and a data body in a key-value pair format.
  • the data domain identifier is used to indicate the category of the structured data to be filtered, where the category is, for example, but not limited to: a CPU usage of the host, an access delay time of a website, etc., and the data domain identifier may be
  • the identification is performed using data or text, and in addition, any manner of identification that can be recognized by the computer can be used as an implementation of the data domain identification and is incorporated herein by reference.
  • the data body of the key-value pair format records detailed information of a key-value pair format (Key-Value format) of the structured data to be filtered, for example, by way of example only, not limited thereto.
  • the information on the left and right sides constitutes the data body of the key-value pair format, here,
  • the key-pairs included in the data body may include one or more, and the number of key-pairs is not limited.
  • the initial to-be-filtered data is obtained from the mass data
  • the first device 11 further includes: a unit for acquiring the initial to-be-filtered data from the distributed message middleware.
  • the first device 11 is distributed message middleware, preferably, the distributed message middleware is MetaQ (a distributed message middleware), MetaQ is a distributed, queue model message middleware, MetaQ It has the following characteristics: it can guarantee strict message order; provide rich message pull mode, efficient subscriber level expansion capability, real-time message subscription mechanism, billion-level message stacking capability, and utilizes MetaQ cluster data Sharding (sharding)
  • the feature allows multiple devices 1 to form multiple peer nodes with identical functions, and enables the cluster to have load balancing capabilities to meet the scalability, high availability, and performance requirements in the context of massive data.
  • the first device 11 may further include: a unit for sending the structured to-be-filtered data to the blocking queue; and correspondingly, the third device 13 includes acquiring the structure from the blocking queue. The unit that is to be filtered.
  • the blocking queue can block further enqueue operations when the queue is full until the queue of the blocking queue is not full.
  • the first device 11 sends the structured to-be-filtered data to the blocking queue, and the structured to-be-filtered data enters a blocking queue, and the third device 13 is configured according to the structured data to be filtered. Waiting for the order, the structured to-be-filtered data is obtained from the blocking queue, and the structured to-be-filtered data is deleted from the blocking queue after being acquired.
  • the blocking queue blocks the operation of the first device 11 to send filtered data into the blocking queue, thereby avoiding insufficient processing capability.
  • the memory usage is too large, so as to play a role of peak clipping and valley filling in the massive data filtering process, to avoid processing failures.
  • the second device 12 is configured to load a filtering rule, where each of the filtering rules includes: a rule domain identifier, a rule name, and a rule operation expression, and establishes an index that is based on the domain identifier of the filtering rule.
  • a first rule list of the filtering rules is configured to load a filtering rule, where each of the filtering rules includes: a rule domain identifier, a rule name, and a rule operation expression, and establishes an index that is based on the domain identifier of the filtering rule.
  • the rule domain identifier is used to indicate the category of the filtering rule, where the category is, for example, and not limited to: CPU usage of the host, access delay time of a website, etc., the rule domain identifier may be adopted. Data, text, etc. are identified, and any manner of identification that can be recognized by the computer can be used as an implementation of the data domain identification and is incorporated herein by reference.
  • the content of the rule domain identifier is the same as or substantially the same as the content of the data domain identifier, so that the third device 13 obtains the data from the first rule list according to the data domain identifier.
  • the domain ID identifies several filtering rules for the rule domain ID.
  • the rule name may be a globally uniquely identified rule name, so as to facilitate the management dimension of the filtering rule. Protection.
  • clusterId hangzhou)&&value>80, rule operation expression It may also include data collection types consisting of non-numeric, string, and other native types, such as (for example only, without limitation): arrays, hash sets, and the like.
  • the second device 12 establishes a first rule list of the filtering rule indexed by the domain identifier of the filtering rule, where the first rule list is used to provide support for the third device 13 to obtain a filtering rule.
  • the third device 13 acquires the structured to-be-filtered data, and obtains, according to the data domain identifier, a plurality of filterings from the first rule list that have rule domain identifiers corresponding to the data domain identifiers. rule. Specifically, the third device 13 acquires, according to the data domain identifier, a plurality of filtering rules having a rule domain identifier corresponding to the data domain identifier from the first rule list.
  • the fourth device 14 performs parallel matching operation on the structured to-be-filtered data by using the acquired plurality of filtering rules.
  • the third device 13 acquires, according to its data domain identifier, a plurality of filtering rules having corresponding identical rule domain identifiers, and the fourth device 14 utilizes each acquired
  • the filtering rule performs a matching operation on the structured data to be filtered
  • the fourth device 14 performs parallel matching calculation on the acquired filtering rules to fully utilize the performance of the multi-core CPU and improve the filtering efficiency.
  • the fourth device 14 includes: a unit for performing rule compilation on the acquired filtering rule to establish a runable abstract syntax tree; and a data body for using the structured data to be filtered as an input parameter Means traversing a number of said runnable abstract syntax trees and utilizing a plurality of said runnable abstract syntax trees for parallel matching calculations.
  • the fourth device 14 implements the function of an abstract syntax tree, and can support arithmetic operations, string operations, relational operations, logical operations, regular expression operations, and set operations, and reserves an extended interface to support user customization. The operation and so on.
  • the fourth device 14 performs rule compilation on the acquired filtering rule to establish an AST (Abstract Syntax Tree), where the abstract syntax tree is an abstract syntax of a regular expression. A tree-like representation of the structure.
  • AST Abstract Syntax Tree
  • the unit for performing rule compilation on the acquired filtering rule to establish a runnable abstract syntax tree includes: analyzing a rule expression of the obtained filtering rule to be converted into an abstract syntax tree a module; and means for pre-computing the abstract syntax tree to obtain the runnable abstract syntax tree.
  • the rule expression of the obtained filtering rule is analyzed to be converted into an abstract syntax tree, which may It is implemented by Antlr (Another Tool for Language Recognition), which can convert user-defined filter rule expression into abstract syntax tree; obtain AST Token stream by lexical analysis of rule expression, Token stream (token) includes analysis Recognized string rules Various arithmetic operators, including but not limited to: operators, numbers, strings, variables, regular expressions, and so on.
  • the arithmetic operators include the following sample code:
  • a rule expression of a filter rule is content in the form of a string:
  • FIG. 8 to FIG. 10 are schematic diagrams showing parallel matching operations of the structured to-be-filtered data by using the acquired plurality of filtering rules according to a specific scenario of the present application.
  • the AST token flow shown in Figure 9 is obtained, and the preservation form in the system uses the operation expression suffix representation to solve the priority problem, as shown in Figure 8, the save form is: OP: Operator, Num: Number, Var: Variable, Regex: Regular Expression, StrArray: String Array.
  • the abstract syntax tree is pre-computed to obtain the runnable abstract syntax tree.
  • the pre-calculation is used to pre-calculate a constant expression in the AST token stream to determine whether the sub-expression is computable, and to check whether each element in the abstract syntax tree is a special type by pre-calculation.
  • the special type of elements are converted into programming language data structure elements, such as, but not limited to, converting the Like operation parameter element interpretation into a regular expression, and converting the In operation parameter element interpretation into a collection.
  • Pre-computation can pre-budget constant expressions in AST to speed up runtime processing, and special types of elements are converted into programming language data structure elements, where the special types of elements are non-numeric, string-shaped Elements of the native type, such as but not limited to data collection types, such as but not limited to arrays, hash maps, hash sets, and the like.
  • the fourth device 14 performs a pre-calculation on the abstract syntax tree shown in FIG. 9 , and the calculation result is an executable abstract syntax tree (AST), wherein the AST token stream As shown in FIG. 10, "0.9”, “Java.util.HashSet['hz', 'qd']” and "Java.util.regex.pattern'AK47 ⁇ W+'” are pre-calculated calculation results. .
  • AST executable abstract syntax tree
  • sample code for the precomputation is as follows:
  • a module for pre-computing the abstract syntax tree to obtain the executable abstract syntax tree is configured to: create a running stack according to the abstract syntax tree, and pass an element in the abstract syntax tree to the running stack, and when the element is an operator, the operator corresponds to two The operand is passed out of the run stack, and the calculation is performed to obtain a calculation result, and when the element is a special element, the special element is converted into a program language data structure element and then passed into the running stack.
  • the fourth device 14 further includes using the data body of the structured data to be filtered as an input parameter, traversing a plurality of the runnable abstract syntax trees, and performing parallel matching calculation by using the plurality of the runnable abstract syntax trees. Unit.
  • the process of performing parallel matching calculation operations using a plurality of said runnable abstract syntax trees is the same as a pre-calculation.
  • all expressions in the AST are computable expressions, so the final calculation result is a certain
  • the value of the value is a boolean value of FALSE or TRUE. If the Boolean value of the calculation result is TRUE, the structured data is determined to satisfy the filtering rule.
  • the device 1 of the present application is assigned 1000 filtering rules, for each of the structured data to be filtered, in the A matching operation is performed concurrently on the 1000 filtering rules in the thread pool of the device 1 to fully utilize the performance of the multi-core CPU to concurrently calculate the filtering rules.
  • the unit for using the data body of the structured data to be filtered as an input parameter, traversing a plurality of the runnable abstract syntax trees, and performing parallel matching calculation by using the plurality of the runnable abstract syntax trees includes: A module for replacing a variable of the runnable abstract syntax tree with a parameter in the data body; a module for performing a matching calculation on the runnable abstract syntax tree with the run stack.
  • a matching calculation is performed on the runnable abstract syntax tree using the run stack, wherein the sample code for processing each node of the AST and putting it into the runtime stack is as follows:
  • the sample code for the corresponding operation on the operator node in the AST is as follows:
  • sample code for matching calculation is as follows:
  • the device 1 can further process the structured data to be filtered, such as an alarm or the like.
  • FIG. 2 is a schematic diagram of a device for filtering data according to a preferred embodiment of the present application.
  • the device 1 includes: a first device 11 ′, a second device 12 ′, a third device 13 ′′, Four devices 14' and a fifth device 15'.
  • the contents of the first device 11', the third device 13', and the fourth device 14' are the same as or substantially the same as those of the first device 11, the third device 13, and the fourth device 14 of the device 1 shown in FIG. For the sake of brevity, we will not repeat them here, but only by reference.
  • the second device 12' is based on the content of the second device 12 shown in FIG. 1, and the second device 12' further includes: establishing an index indexed according to the rule of the filtering rule a unit of the second rule list of the filtering rule; the second device 12' establishes a first rule list and a second rule according to the rule domain identifier of the filtering rule and the rule name of the filtering rule as an index of two dimensions a list, wherein the first rule list indexed by the rule domain identifier of the filtering rule is used for searching when filtering data, and the rule name of the filtering rule is The second rule list is used for searching and managing the filter rules.
  • the filtering rule in the first rule list is searched according to the data domain identifier matching, the corresponding filtering rule list is found, and the filtering rule list is traversed, and the data body of the data to be filtered is formatted.
  • a concurrent match calculation is performed for each rule in the list.
  • the second rule list facilitates management of the filtering rules.
  • the fifth device 15' is configured to add a filtering rule, delete a filtering rule, or modify and compile an existing filtering rule.
  • the fifth device 15' includes a unit for adding a newly added filtering rule to the second rule list; a unit for deleting a corresponding filtering rule from the second rule list; Searching for the filtering rule from the second rule list, and modifying and compiling the searched filter rule.
  • the fifth device 15' can modify, add, and delete the filtering rules to improve the flexibility of the filtering rules.
  • FIG 3 is a schematic diagram of an apparatus for filtering data according to another preferred embodiment of the present application, wherein the apparatus 1 includes a first device 11", a second device 12", and a third device 13" The fourth device 14", the fifth device 15" and the sixth device 16".
  • first device 11 the second device 12
  • the third device 13 the fourth device 14
  • the fifth device 15 the first device 11'
  • the contents of the device 12', the third device 13', the fourth device 14', and the fifth device 15" are the same or substantially the same, and are not described again for brevity and are hereby incorporated by reference.
  • each of the filtering rules further includes: information of the notifier to which the filtering rule is bound; the sixth device 16' is configured to send the structured to-be-filtered data that meets the corresponding filtering rule to the Filters the notifiers bound to the rules for transmission.
  • the notifier is a set of implementations of the reservation notification interface, and can implement a customized notification manner, such as using different transmission protocols, different compression algorithms, and different serialization algorithms to transmit to the downstream system cluster. In a different system. The notifier can perform free combination assembly binding to any filtering rule when the filtering rule is created.
  • step S11 shows a flow chart of a method for filtering data according to an aspect of the present application, wherein the method includes: step S11, step S12, step S13, and step S14.
  • the step S11 includes: acquiring initial data to be filtered, and converting the initial data to be filtered into structured data to be filtered, where the structured data to be filtered includes a data domain identifier and a key-value pair format.
  • the step S12 includes: loading a filter rule, wherein each of the filter rules includes a rule domain identifier, a rule name, a rule operation expression, and establishing the index with the domain identifier of the filter rule as an index Filtering the first rule list of the rule;
  • the step S13 includes: acquiring the structured data to be filtered, and according to the number Obtaining, according to the domain identifier, a plurality of filtering rules having a rule domain identifier corresponding to the data domain identifier from the first rule list;
  • the step S14 includes: using the obtained plurality of filtering rules to filter the structured to be filtered
  • the data is subjected to parallel matching operations.
  • the initial data to be filtered is acquired, and the initial data to be filtered is converted into structured data to be filtered, wherein the structured data to be filtered includes a data domain identifier and a key-value pair.
  • Formatted Data Body the structured data to be filtered includes a data field identifier and a data body in a key-value pair format.
  • the data domain identifier is used to indicate the category of the structured data to be filtered, where the category is, for example, but not limited to: a CPU usage of the host, an access delay time of a website, etc., and the data domain identifier may be
  • the identification is performed using data or text, and in addition, any manner of identification that can be recognized by the computer can be used as an implementation of the data domain identification and is incorporated herein by reference.
  • the data body of the key-value pair format records detailed information of a key-value pair format (Key-Value format) of the structured data to be filtered, for example, by way of example only, not limited thereto.
  • the initial to-be-filtered data is obtained from the massive data
  • the step S11 further includes: obtaining the initial to-be-filtered data from the distributed message middleware, by using a distributed message middleware, preferably, MetaQ (one Distributed message middleware) is a distributed, queue-based message middleware that has the following characteristics: it can guarantee strict message order; provide rich message pull mode, efficient subscriber level expansion capability, real-time message a subscription mechanism, a billion-level message stacking capability, utilizing the characteristics of MetaQ's cluster data sharding, and FIG. 7 illustrates a system for applying the data filtering device according to a preferred embodiment of the present application. Schematic diagram of the device. Multiple devices 1 form multiple peer nodes with the same function to cluster, and enable the cluster to have load balancing capability to meet the scalability, high availability and performance requirements in the context of massive data.
  • MetaQ one Distributed message middleware
  • the step S11 further includes: sending the structured to-be-filtered data to the blocking queue; and correspondingly, the step S13 includes: acquiring the structured to-be-filtered data from the blocking queue.
  • the blocking queue can block further enqueue operations when the queue is full until the queue of the blocking queue is not full.
  • the step S11 sends the structured data to be filtered to the blocking queue, and the structured data to be filtered enters the blocking queue to wait, and the step S13 is performed according to the waiting order of the structured data to be filtered.
  • the obtaining the structured to-be-filtered data from the blocking queue is deleted from the blocking queue when the structured to-be-filtered data is obtained.
  • the structured data to be filtered waiting in the blocking queue is full.
  • the blocking queue blocks the operation of sending the filtered data into the blocking queue in step S11, so that the memory usage is too large when the processing capability is insufficient, so that the peak data is filled in the massive data filtering process. Role to avoid handling failures.
  • the filtering rule is loaded, where each of the filtering rules includes: a rule domain identifier, a rule name, and a rule operation expression, and establishes an index based on the domain identifier of the filtering rule.
  • a list of first rules for filtering rules is loaded, where each of the filtering rules includes: a rule domain identifier, a rule name, and a rule operation expression, and establishes an index based on the domain identifier of the filtering rule.
  • the rule domain identifier is used to indicate the category of the filtering rule, where the category is, for example, and not limited to: CPU usage of the host, access delay time of a website, etc., the rule domain identifier may be adopted. Data, text, etc. are identified, and any manner of identification that can be recognized by the computer can be used as an implementation of the data domain identification and is incorporated herein by reference.
  • the content of the rule domain identifier is the same as or substantially the same as the content of the data domain identifier, so that the step S13 obtains the identifier with the data domain from the first rule list according to the data domain identifier.
  • the rule name may be a globally uniquely identified rule name to facilitate management and maintenance of the filtering rule.
  • clusterId hangzhou)&&value>80, rule operation expression It may also include data collection types consisting of non-numeric, string, and other native types, such as (for example only, without limitation): arrays, hash sets, and the like.
  • step S12 includes: establishing a first rule list of the filtering rule indexed by the domain identifier of the filtering rule, where the first rule list is used to provide support for obtaining the filtering rule in step S13.
  • the structured to-be-filtered data is acquired, and the rule domain identifiers corresponding to the data domain identifiers are obtained from the first rule list according to the data domain identifiers.
  • Filter rules Specifically, the step S13 acquires, according to the data domain identifier, a plurality of filtering rules having a rule domain identifier corresponding to the data domain identifier from the first rule list.
  • the structured to-be-filtered data is subjected to a parallel matching operation by using the acquired plurality of filtering rules.
  • the step S13 obtains a plurality of filtering rules having corresponding rule domain identifiers according to the data domain identifier, and step S14 uses each of the acquired filtering rule pairs.
  • the structured data to be filtered is subjected to a matching operation.
  • the step S14 performs parallel matching calculation on a plurality of acquired filtering rules to fully utilize the performance of the multi-core central processing unit and improve filtering efficiency.
  • the step S14 includes: performing rule compilation on the acquired filtering rule to establish a runnable pumping Like a syntax tree; the data body of the structured data to be filtered is used as an input parameter, a plurality of the runnable abstract syntax trees are traversed, and a plurality of the runnable abstract syntax trees are used for parallel matching calculation.
  • the step S14 implements the function of the abstract syntax tree, and can support arithmetic operations, string operations, relational operations, logical operations, regular expression operations, and set operations, and reserves an extended interface to support user-defined operations. Wait.
  • the obtained filter rule is subjected to rule compilation to establish an executable abstract syntax tree (AST, Abstract Syntax Tree), where the abstract syntax tree is a tree-like representation of an abstract syntax structure of the regular expression. .
  • AST Abstract Syntax Tree
  • Performing rule compilation on the obtained filtering rule to establish a runnable abstract syntax tree includes: analyzing the rule expression of the obtained filtering rule to be converted into an abstract syntax tree, specifically, Antlr ( Another Tool for Language Recognition) can convert a user-defined filter rule expression into an abstract syntax tree; obtain a Token stream of the AST by lexical analysis of the rule expression, and the Token stream (token) includes the analysis and identification.
  • Antlr Another Tool for Language Recognition
  • Token includes the analysis and identification.
  • String rules Various arithmetic operators, including but not limited to, for example: operators, numbers, strings, variables, regular expressions, and so on.
  • the example code of the operation operator is the same as or substantially the same as the example code of the operation operator of the abstract syntax tree converted by the fourth device 14 of the device 1 shown in FIG. 1. For the sake of brevity, no further details are provided. The way is included here.
  • a rule expression of a filter rule is content in the form of a string:
  • the AST token flow shown in Figure 9 is obtained, and the preservation form in the system uses the operation expression suffix representation to solve the priority problem, as shown in Figure 8, the save form is: OP: Operator, Num: Number, Var: Variable, Regex: Regular Expression, StrArray: String Array.
  • the abstract syntax tree is pre-computed to obtain the runnable abstract syntax tree.
  • the pre-calculation is used to pre-calculate a constant expression in the AST token stream to determine whether the sub-expression is computable, and to check whether each element in the abstract syntax tree is a special type by pre-calculation.
  • the special type of elements are converted into programming language data structure elements, such as, but not limited to, converting the Like operation parameter element interpretation into a regular expression, and converting the In operation parameter element interpretation into a collection.
  • Pre-computation can pre-budget constant expressions in AST to speed up runtime processing, and special types of elements are converted into programming language data structure elements, where the special types of elements are non-numeric, string-shaped Elements of the native type, such as but not limited to data collection types, such as but not limited to arrays, hash maps, hash sets, and the like.
  • the abstract syntax tree shown in FIG. 9 is pre-calculated, and the calculation result is an executable abstract syntax tree (AST), wherein the token flow of the AST is as shown in FIG. Among them, "0.9”, “Java.util.HashSet['hz', 'qd']” and “Java.util.regex.pattern'AK47 ⁇ W+'” are pre-computed calculation results.
  • AST executable abstract syntax tree
  • the example code for performing the pre-calculation may be the same as or substantially the same as the content of the pre-computed example code shown in FIG. 1 for the sake of brevity, and will not be described again for brevity, and is only included herein by reference.
  • pre-computing the abstract syntax tree includes:
  • the data body of the structured data to be filtered is used as an input parameter, traversing a plurality of the runnable abstract syntax trees, and using a plurality of the runnable abstract syntax trees to perform parallel matching calculation, using a plurality of The process of running the abstract syntax tree for parallel matching calculations is the same as a pre-computation. Under normal circumstances, all expressions in the AST are computable expressions, so the final result is a certain value. If the Boolean value is FALSE or TRUE, if the Boolean value of the calculation result is TRUE, the structured data is judged to satisfy the filtering rule.
  • the parallel matching calculation using a plurality of the runnable abstract syntax trees performs matching calculation on the data by using the runnable abstract syntax tree, for example, when the device 1 described in the present application is assigned 1000 filters. For each of the structured data to be filtered, a matching operation is performed concurrently on the 1000 filtering rules in the thread pool of the device 1 to fully utilize the performance of the multi-core CPU to concurrently calculate the filtering rules.
  • performing parallel matching calculation using a plurality of the runnable abstract syntax trees includes: replacing a variable of the runnable abstract syntax tree with a parameter in the data body; using the run stack to perform the runnable abstract syntax The tree performs matching calculations.
  • the example code for replacing the variable of the runnable abstract syntax tree with the parameter in the data body is the same as or substantially the same as the content of the fourth device 14 of the device 1 in FIG. 1 replacing the example code, for the sake of brevity, I will not repeat them here, but I will only include them here by reference.
  • the example code for performing the corresponding operation on the operator node in the AST is the same as or substantially the same as the example code of the corresponding operation of the fourth device 14 of the device 1 in FIG. 1, and will not be described again for the sake of brevity. It is included here by reference only.
  • the content of the example code for performing the matching calculation with the fourth device 14 of the device 1 in FIG. 1 is the same or substantially the same, and is not described again for brevity, and is only included herein by reference.
  • the method may further process the structured data to be filtered, such as an alarm or the like.
  • FIG. 5 is a flow chart showing a method for filtering data according to a preferred embodiment of the present application, the method comprising: step S11', step S12', step S13', step S14' and step S15'.
  • step S11', the step S13' and the step S14' are the same as or substantially the same as the contents of the step S11, the step S12 and the step S14 shown in FIG. 4, and are not described again for brevity, and are only included by reference. herein.
  • the step S12' is based on the content of the step S12 shown in FIG. 4, and the step S12' further includes: establishing a second of the filtering rule indexed according to the rule name of the filtering rule. a rule list; the step S12' establishes a first rule list and a second rule list according to the rule domain identifier of the filtering rule and the rule name of the filtering rule, wherein the filtering rule is The rule list identifies the first rule list of the index for searching when the data is filtered, and the second rule list with the rule name of the filter rule as the index for the management and maintenance of the filter rule.
  • the filtering rule in the first rule list is searched according to the data domain identifier matching, the corresponding filtering rule list is found, and the filtering rule list is traversed, and the data body of the data to be filtered is formatted.
  • a concurrent match calculation is performed for each rule in the list.
  • the second rule list facilitates management of the filtering rules.
  • a filtering rule is added, a filtering rule is deleted, or an existing filtering rule is modified and compiled.
  • the step S15' includes at least one of: adding a new filtering rule to the second rule list; deleting a corresponding filtering rule from the second rule list; from the second
  • the filter rule is searched in the rule list, and the searched filter rule is modified and compiled.
  • the step S15' can modify, add, and delete the filter rule to improve the flexibility of the filter rule.
  • step S11 is a flowchart of a method for filtering data according to another preferred embodiment of the present application, where the method includes step S11", step S12", step S13", step S14", step S15" And step S16".
  • step S11 The content of the step S11", the step S12", the step S13", the step S14” and the step S15” and the step S11', the step S12', the step S13', the step S14' and the step S15” shown in Fig. 5
  • each of the filtering rules further includes: information of the notifier to which the filtering rule is bound; in the step S16', sending the structured to-be-filtered data that satisfies the corresponding filtering rule to the filtering
  • the notifier bound to the rule for transmission is a set of implementations of the reservation notification interface, which can implement a customized communication. Knowing methods, such as using different transport protocols, different compression algorithms, and different serialization algorithms are transmitted to different systems in the downstream system cluster. The notifier can perform free combination assembly binding to any filtering rule when the filtering rule is created.
  • the device and method for data filtering adopts a streaming operation mode, which does not cache or not solidify data in the memory, that is, each time the initial data to be filtered is acquired.
  • the initial data to be filtered is converted into structured data to be filtered, and the matching calculation is performed in real time by using the corresponding filtering rule, and then the filtering result is obtained, and the real-time problem of filtering the massive streaming data is solved;
  • the method and device for filtering data according to an apparatus and method for data filtering support arithmetic operations, string operations, relational operations, logical operations, regular expression operations, set operations,
  • the extended interface is reserved, and the filtering rule is a simple operation expression with a variable, which solves the problem that the filtering rule description is complicated, the expansion is difficult, and the management is difficult;
  • the device and method for data filtering provided by the present application according to an embodiment of the present application are independently designed and developed, the cost is relatively low, and can be monitored and tuned on any code path.
  • the performance indicators are roughly as follows: a single 4-core 8G virtual machine can support 500,000 filter rules, the processing flow data TPS reaches 20000, the filtered effective data TPS reaches 2000, and the average system load is stable. Around load1-4, CPU resources are effectively utilized.
  • the terminal, the device of the service network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media.
  • Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • the present application can be implemented in software and/or a combination of software and hardware, for example, Implemented by an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device.
  • the software program of the present application can be executed by a processor to implement the steps or functions described above.
  • the software programs (including related data structures) of the present application can be stored in a computer readable recording medium such as a RAM memory, a magnetic or optical drive or a floppy disk and the like.
  • some of the steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.
  • a portion of the present application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide a method and/or technical solution in accordance with the present application.
  • the program instructions for invoking the method of the present application may be stored in a fixed or removable recording medium, and/or transmitted by a data stream in a broadcast or other signal bearing medium, and/or stored in a The working memory of the computer device in which the program instructions are run.
  • an embodiment in accordance with the present application includes a device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, triggering
  • the apparatus operates based on the aforementioned methods and/or technical solutions in accordance with various embodiments of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种用于过滤数据的设备及方法，每次获取初始待过滤数据后将所述初始待过滤数据转换为结构化待过滤数据，并利用相应的过滤规则实时进行匹配计算，随即得到过滤结果，解决实时性问题，同时支持算术运算，字符串运算，关系运算，逻辑运算，正则表达式运算，集合运算，且预留了扩展接口，并且所述过滤规则为简单的带变量的运算表达式形式，解决了过滤规则描述复杂、扩展不易及管理困难的问题。

Description

一种用于过滤数据的设备及方法
本申请要求2015年07月13日递交的申请号为201510408180.1、发明名称为“一种用于过滤数据的设备及方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机领域，尤其涉及一种从海量数据中根据设定的过滤规则实时过滤出满足过滤规则的数据的技术。
背景技术
随着信息技术爆发式增长，数据量与日俱增，众多领域对海量数据的处理的要求不断增加。
针对如何从海量数据中根据设定的过滤规则过滤出满足过滤规则的数据，现有技术中有以下几种方法：
基于内存关系型数据库的SQL语句(Structured Query Language)来过滤有效数据，然而，该方法需要将海量数据缓存在内存数据库的逻辑数据表中，占用大量内存资源，且SQL语句的周期性执行难以达到实时性要求；
基于Hbase(一个分布式的、面向列的开源数据库)的海量数据存储方案，使用Map-Reduce算法(一种编程模型算法，用于大规模数据集的并行运算)来过滤有效数据，然而，Map-Reduce模型任务是类似于批处理的后计算模式，对已经存储在Hbase中的海量数据，只能周期性的执行运算匹配结果，实时性难以得到保障，并且复杂的Map-Reduce模型任务需要通过扩展编写来实现，难以满足对大量过滤规则的实时可变和多种计算的需求；
基于CEP引擎(复杂事件处理，Complex Event Processing)，使用模式匹配算法来过滤有效数据更适合做企业应用系统的监测和决策控制，然而成熟的CEP引擎大多是商业软件，用户成本高，并且CEP引擎都有各自的模式规则描述方法，比如Drools使用XML格式，Esper使用EPL格式，针对不同的系统的需求需要编写大量的适配代码来使用，且对于非标准化的匹配算法需要扩展编写来实现，增加实现难度，此外，CEP引擎的实现各不相同，故不易对CEP引擎的性能监测和调优。
发明内容
本申请要解决的技术问题是如何能够在不占用大量内存资源下，根据设定的过滤规则，从海量数据中实时地过滤出满足过滤规则，并且能够满足大量过滤规则的实时可变和多种计算的需求。
为实现上述目的，本申请提供了一种用于过滤数据的方法，其中，所述方法包括：
获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；
加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；
获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；
利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
进一步地，所述获取初始待过滤数据包括：
从分布式消息中间件中获取所述初始待过滤数据。
进一步地，将所述初始待过滤数据转换为结构化待过滤数据还包括：
将所述结构化待过滤数据发送至阻塞队列；
获取所述结构化待过滤数据包括：
从所述阻塞队列中获取所述结构化待过滤数据。
进一步地，利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算包括：
对所获取的过滤规则进行规则编译，以建立可运行抽象语法树；
将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算。
进一步地，对所获取的过滤规则进行规则编译，以建立可运行抽象语法树包括：
对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树；
对所述抽象语法树进行预计算，以获得所述可运行抽象语法树；
其中，对所述抽象语法树进行一次预计算包括：
根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中；
当所述元素为操作符时，将所述操作符对应的两个操作数传出所述运行堆栈，计算以获得计算结果；
当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入运行堆栈中。
进一步地，利用若干所述可运行抽象语法树进行并行匹配计算包括：
将所述可运行抽象语法树的变量替换为所述数据体中的参数；
利用所述运行堆栈对所述可运行抽象语法树进行匹配计算。
进一步地，所述方法还包括：
新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
进一步地，建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表还包括：
建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表；
所述新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译包括至少以下任一项：
将新增的过滤规则增加至所述第二规则列表中；
从所述第二规则列表中删除相应的过滤规则；
从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译。
进一步地，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；
所述方法还包括：
将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。
根据本申请另一方面还提供了一种用于过滤数据的设备，其中，所述设备包括：
第一装置，用于获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；
第二装置，用于加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；
第三装置，用于获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；
第四装置，用于利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配 运算。
进一步地，所述第一装置包括：
从分布式消息中间件中获取所述初始待过滤数据的单元。
进一步地，所述第一装置包括：
用于将所述结构化待过滤数据发送至阻塞队列的单元；
所述第三装置包括：
从所述阻塞队列中获取所述结构化待过滤数据的单元。
进一步地，所述第四装置包括：
用于对所获取的过滤规则进行规则编译，以建立可运行抽象语法树的单元；
用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元。
进一步地，所述用于对所获取的过滤规则进行规则编译，以建立可运行抽象语法树的单元包括：
用于对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树的模块；
用于对所述抽象语法树进行预计算，以获得所述可运行抽象语法树的模块，其中，该模块用于：
根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中，
当所述元素为操作符时，将操作符对应的两个操作数传出所述运行堆栈，计算以获得计算结果，
用于当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入所述运行堆栈中。
进一步地，所述用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元包括：
用于将所述可运行抽象语法树的变量替换为所述数据体中的参数的模块；
用于利用所述运行堆栈对所述可运行抽象语法树进行匹配计算的模块。
进一步地，所述设备还包括：
第五装置，用于新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
进一步地，所述第二装置还包括：
建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表的单元；
所述第五装置包括：
用于将新增的过滤规则增加至所述第二规则列表中的单元；
用于从所述第二规则列表中删除相应的过滤规则的单元；
用于从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译的单元。
进一步地，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；
所述设备还包括：
第六装置，用于将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。
与现有技术相比，根据本申请一实施例所提供的用于数据过滤的设备及方法采用流式运算方式，不会在内存中缓存也不会固化数据，即每次获取初始待过滤数据后将所述初始待过滤数据转换为结构化待过滤数据，并利用相应的过滤规则实时进行匹配计算，随即得到过滤结果，解决海量流式数据的过滤的实时性问题；
进一步地，根据本申请一实施例所提供的用于数据过滤的设备及方法所述过滤数据的方法和设备支持算术运算，字符串运算，关系运算，逻辑运算，正则表达式运算，集合运算，且预留了扩展接口，并且所述过滤规则为简单的带变量的运算表达式形式，解决了过滤规则描述复杂、扩展不易及管理困难的问题；
此外，本申请根据本申请一实施例所提供的用于数据过滤的设备及方法为自主设计开发，成本相对较低，且在任意代码路径上都可以监控和调优。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：
图1示出根据本申请一方面提供的一种用于过滤数据的设备的设备示意图；
图2示出根据本申请一优选的实施例提供的一种用于过滤数据的设备的设备示意图；
图3示出根据本申请另一优选的实施例提供的一种用于过滤数据的设备的设备示意图；
图4示出根据本申请一方面提供的一种用于过滤数据的方法流程图；
图5示出根据本申请一优选的实施例提供的一种用于过滤数据的方法流程图；
图6示出根据本申请另一优选的实施例提供的一种用于过滤数据的方法流程图；
图7示出根据本申请一优选的实施例提供的一种包括所述用于过滤数据设备的系统的设备示意图；
图8至图10示出根据本申请一具体场景中利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算的示意图。
附图中相同或相似的附图标记代表相同或相似的部件。
具体实施方式
下面结合附图对本申请作进一步详细描述。
图1示出根据本申请一方面提供的一种用于过滤数据的设备的设备示意图，其中，所述设备1包括：第一装置11、第二装置12、第三装置13和第四装置14。
具体地，所述第一装置11用于获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；所述第二装置12用于加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称、规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；所述第三装置13用于获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；所述第四装置14用于利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
进一步地，所述第一装置11用于获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体在此，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体。
其中，所述数据领域标识用于表明所述结构化待过滤数据的类别，其中，所述类别例如且不限于：主机的CPU占用率、某网站的访问延迟时间等，所述数据领域标识可以采用数据或文字等进行标识，此外，任何能够被计算机识别的标识的方式都可以作为所述数据领域标识的实施方式，并以引用的方式包含于此。其中，所述键-值对格式的数据体记录所述结构化待过滤数据的键-值对格式(Key-Value格式)的详细信息，所述数据体例如(仅为举例，并不限于此)：instanceId＝AY123456，clusterId＝Hangzhou，value＝92，bizTime＝1427041923825,unit＝Percent，其中，每一等号左侧表示键(Key)，每一等号右侧表示值(Value)，等号左右两侧的信息组成键-值对格式的数据体，在此， 所述数据体所包括的键-对可包括一个或多个，其键-对的数量并不受限制。
优选的，所述初始待过滤数据从海量数据中获取，所述第一装置11还包括：从分布式消息中间件中获取所述初始待过滤数据的单元。所述第一装置11通过分布式消息中间件，优选的，所述分布式消息中间件为MetaQ(一种分布式消息中间件)，MetaQ是一款分布式、队列模型的消息中间件，MetaQ具有以下特点：能够保证严格的消息顺序；提供丰富的消息拉取模式，高效的订阅者水平扩展能力，实时的消息订阅机制，亿级消息堆积能力，利用了MetaQ的集群数据Sharding(分片)的特性，可以使多个设备1形成多个功能完全相同的对等节点进行集群，并使使集群具备了负载均衡能力，满足在海量数据背景下的可扩展性，高可用性和性能要求。
优选的，所述第一装置11还可以包括：用于将所述结构化待过滤数据发送至阻塞队列的单元；相应地，所述第三装置13包括从所述阻塞队列中获取所述结构化待过滤数据的单元。
在此，所述阻塞队列能够在队列满的时候阻塞进一步的入队操作直至所述阻塞队列的队列不满。具体地，所述第一装置11将所述结构化待过滤数据发送至阻塞队列，则所述结构化待过滤数据进入阻塞队列等待，所述第三装置13根据所述结构化待过滤数据的等待次序，依此从所述阻塞队列中获取所述结构化待过滤数据，当所述结构化待过滤数据被获取后即从所述阻塞队列删除。在此，所述阻塞队列中等待的结构化待过滤数据占满所述阻塞队列时，所述阻塞队列阻塞所述第一装置11发送过滤数据进入阻塞队列的操作，从而能够避免处理能力不足时，内存占用过大，从而在海量数据过滤过程中起到削峰填谷的作用，避免处理故障。
进一步地，所述第二装置12用于加载过滤规则，其中，每一所述过滤规则包括：规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表。
在此，所述规则领域标识用于表明所述过滤规则的类别，其中，所述的类别例如且不限于：主机的CPU占用率、某网站的访问延迟时间等，所述规则领域标识可以采用数据或文字等进行标识，此外，任何能够被计算机识别的标识的方式都可以作为所述数据领域标识的实施方式，并以引用的方式包含于此。优选地，所述规则领域标识的内容与所述数据领域标识的内容相同或基本相同，以便所述第三装置13根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则。其中，所述规则名称可以是全局唯一识别的规则名称，以便于过滤规则的管理维 护。其中，规则运算表达式可以是数字、字符串形式组成的规则表达式，例如(仅为举例，并不限于此)：instanceId＝'AY123456'||clusterId＝hangzhou)&&value>80，规则运算表达式还可以包括非数字、字符串等原生类型组成的数据集合类型，例如(仅为举例，并不限于此)：数组、哈希集合等。
进一步地，所述第二装置12建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表，所述第一规则列表用于为第三装置13获取过滤规则提供支持。
进一步地，所述第三装置13获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则。具体地，所述第三装置13根据所述数据领域标识从所述第一规则列表中获取具有与数据领域标识相对应的规则领域标识的若干过滤规则。
进一步地，所述第四装置14利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
优选的，对于每一所述结构化待过滤数据，所述第三装置13根据其数据领域标识获取若干具有相应相同的规则领域标识的若干过滤规则，则第四装置14利用每一所获取的过滤规则对所述结构化待过滤数据进行一次匹配运算，所述第四装置14对若干所获取的过滤规则进行并行匹配计算，以充分利用多核中央处理器的性能，提高过滤效率。
具体地，所述第四装置14包括：用于对所获取的过滤规则进行规则编译，以建立可运行抽象语法树的单元；和用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元。
所述第四装置14实现了抽象语法树的功能，能够支持算术运算，字符串运算，关系运算，逻辑运算，正则表达式运算和集合运算等，并预留了扩展接口，可支持用户自定义的运算等。
进一步地，所述第四装置14对所获取的过滤规则进行规则编译，以建立可运行抽象语法树(AST，Abstract Syntax Tree)，在此，所述抽象语法树是是规则表达式的抽象语法结构的树状表现形式。
具体地，所述用于对所获取的过滤规则进行规则编译、以建立可运行抽象语法树的单元包括：用于对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树的模块；和用于对所述抽象语法树进行预计算，以获得所述可运行抽象语法树的模块。
具体地，对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树，可以 采用Antlr(Another Tool for Language Recognition)来实现，能够将用户自定义的过滤规则表达式转化为抽象语法树；通过对规则表达式进行词法分析得到AST的Token流，Token流(令牌)包括分析识别出的字符串规则各种运算操作符，运算操作符包括但不限于例如：操作符、数字、字符串、变量、正则表达式等。
其中，运算操作符例如包括以下示例代码：
Figure PCTCN2016088302-appb-000001
Figure PCTCN2016088302-appb-000002
在具体的应用场景中，例如过滤规则的规则表达式为以下字符串形式的内容：
CPU>90/100and clusterId in[‘hz’,’qd’]and instanceId like‘AK47\w+’
图8至图10示出根据本申请一具体场景中利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算的示意图。通过编写Antlr词法分析规则，得到如图9所示的AST token流，并在系统中的保存形式采用运算表达式后缀表示法来解决优先级问题，如图8所示，保存形式为：OP:操作符，Num:数字，Var:变量，Regex:正则表达式，StrArray:字符串数组。
接着，对所述抽象语法树进行预计算，以获得所述可运行抽象语法树。其中，所述预计算用于将AST token流中的常量表达式进行预先计算，以判断子表达式是否可计算，并通过预计算检查所述抽象语法树中每个元素是否为特殊类型，将其中的特殊类型的元素转换为程序语言数据结构元素，例如但不限于，将Like操作参数元素解释转化为正则表达式，将In操作参数元素解释转化为集合。通过预计算能够将AST中的常量表达式进行预先预算，从而加快运行时处理速度，并特殊类型的元素转化为程序语言数据结构元素，其中，所述特殊类型的元素是非数字、字符串组成的原生类型的元素，例如但不限于数据集合类型，例如但不限于数组、哈希map、哈希集合等。
接上例，在具体的场景中，所述第四装置14对图9所示的所述抽象语法树进行一次预计算，计算结果为可运行抽象语法树(AST)，其中，AST的token流如图10所示，其中，“0.9”、“Java.util.HashSet[‘hz’，‘qd’]”及“Java.util.regex.pattern‘AK47\W+’”为经过预计算的计算结果。
在一可选的实施例中，进行预计算的示例代码如下：
Figure PCTCN2016088302-appb-000003
Figure PCTCN2016088302-appb-000004
当然本领域技术人员应当能够理解，上述示例代码仅为举例，今后可能出现的进行预计算的方法、代码等其他形式，如适用本申请，均能够以引用的方式包含于本申请的保护范围之内。
具体地，用于对所述抽象语法树进行预计算，以获得所述可运行抽象语法树的模块， 其中，该模块用于：根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中，当所述元素为操作符时，将操作符对应的两个操作数传出所述运行堆栈，计算以获得计算结果，用于当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入所述运行堆栈中。
进一步地，所述第四装置14还包括将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元。
利用若干所述可运行抽象语法树进行并行匹配计算运算的过程与一趟预计算相同，在正常情况下运行时AST中的所有表达式均为可计算表达式，所以最终的计算结果是一个确定的值，该值为布尔值FALSE或者TRUE，如果计算结果的布尔值为TRUE，则该结构化数据则判断为满足该过滤规则。
在此，通过利用所述可运行抽象语法树对所述数据进行匹配计算，当本申请所述设备1被分配到了1000个过滤规则，则对于每一所述结构化待过滤数据，在所述设备1的线程池中对这1000个过滤规则并发执行匹配运算，以充分利用多核CPU的性能来并发计算过滤规则。
具体地，所述用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元包括：用于将所述可运行抽象语法树的变量替换为所述数据体中的参数的模块；用于利用所述运行堆栈对所述可运行抽象语法树进行匹配计算的模块。
其中，将所述可运行抽象语法树的变量替换为所述数据体中的参数的示例代码如下：
Figure PCTCN2016088302-appb-000005
利用所述运行堆栈对所述可运行抽象语法树进行匹配计算，其中，对AST每个节点进行处理，并放入运行时栈的示例代码如下：
Figure PCTCN2016088302-appb-000006
其中，对AST中的操作符节点进行相应的运算的示例代码如下：
Figure PCTCN2016088302-appb-000007
Figure PCTCN2016088302-appb-000008
其中，进行匹配计算的示例代码如下：
Figure PCTCN2016088302-appb-000009
当然本领域技术人员应当能够理解，上述示例代码仅为举例，今后可能出现的方法、代码等其他形式，如适用本申请，均能够以引用的方式包含于本申请的保护范围之内。
此后，所述设备1可以对该结构化待过滤数据进行进一步处理，例如报警等。
图2示出根据本申请一优选实施例提供的一种用于过滤数据的设备的设备示意图，所述设备1包括：第一装置11’、第二装置12’、第三装置13’、第四装置14’和第五装置15’。
所述第一装置11’、第三装置13’和第四装置14’的内容与图1所示的设备1的第一装置11、第三装置13和第四装置14的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
优选的，所述第二装置12’在引用图1所示的第二装置12的内容的基础上，所述第二装置12’还包括：建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表的单元；所述第二装置12’按照所述过滤规则的规则领域标识和所述过滤规则的规则名称为两个维度的索引建立第一规则列表和第二规则列表，其中，以所述过滤规则的规则领域标识为索引的第一规则列表供过滤数据时查找，以所述过滤规则的规则名称为索 引的第二规则列表供过滤规则的管理和维护时查找。获得所述结构化待过滤数据时，根据数据领域标识匹配查找第一规则列表中的过滤规则，找到对应的过滤规则的列表，并遍历该过滤规则的列表，将格式化待过滤数据的数据体作为输入参数，对列表中的每一规则做并发匹配计算。所述第二规则列表便于对过滤规则进行管理。
所述第五装置15’用于新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
具体地，所述第五装置15’包括用于将新增的过滤规则增加至所述第二规则列表中的单元；用于从所述第二规则列表中删除相应的过滤规则的单元；用于从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译的单元。所述第五装置15’能够对过滤规则进行修改和增删操作，提高过滤规则的灵活性。
图3示出根据本申请另一优选实施例提供的一种用于过滤数据的设备的设备示意图，其中，所述设备1包括第一装置11”、第二装置12”、第三装置13”、第四装置14”、第五装置15”和第六装置16”。
其中，所述第一装置11”、第二装置12”、第三装置13”、第四装置14”和第五装置15”与图2所示的设备1的第一装置11’、第二装置12’、第三装置13’、第四装置14’和第五装置15”的内容相同或基本相同，为简明起见，不再赘述，并以引用的方式包含于此。
在此，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；所述第六装置16’用于将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。其中，在此，所述通知器是对预留通知接口的一组实现，可实现自定义的通知方式，比如使用不同的传输协议，不同的压缩算法，不同的序列化算法传输至下游系统集群中不同的系统中。其中，所述通知器可以在过滤规则创建时，即进行自由地组合装配绑定至任意过滤规则。
图4示出根据本申请一方面提供的一种用于过滤数据的方法流程图，其中，所述方法包括：步骤S11、步骤S12、步骤S13和步骤S14。
具体地，所述步骤S11包括：获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；所述步骤S12包括：加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称、规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；所述步骤S13包括：获取所述结构化待过滤数据，并根据所述数 据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；所述步骤S14包括：利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
进一步地，在所述步骤S11中：获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体在此，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体。
其中，所述数据领域标识用于表明所述结构化待过滤数据的类别，其中，所述类别例如且不限于：主机的CPU占用率、某网站的访问延迟时间等，所述数据领域标识可以采用数据或文字等进行标识，此外，任何能够被计算机识别的标识的方式都可以作为所述数据领域标识的实施方式，并以引用的方式包含于此。其中，所述键-值对格式的数据体记录所述结构化待过滤数据的键-值对格式(Key-Value格式)的详细信息，所述数据体例如(仅为举例，并不限于此)：instanceId＝AY123456，clusterId＝Hangzhou，value＝92，bizTime＝1427041923825,unit＝Percent，其中，每一等号左侧表示键(Key)，每一等号右侧表示值(Value)，等号左右两侧的信息组成键-值对格式的数据体，在此，所述数据体所包括的键-对可包括一个或多个，其键-对的数量并不受限制。
优选的，所述初始待过滤数据从海量数据中获取，所述步骤S11还包括：从分布式消息中间件中获取所述初始待过滤数据，通过分布式消息中间件，优选的，MetaQ(一种分布式消息中间件)是一款分布式、队列模型的消息中间件，具有以下特点：能够保证严格的消息顺序；提供丰富的消息拉取模式，高效的订阅者水平扩展能力，实时的消息订阅机制，亿级消息堆积能力，利用了MetaQ的集群数据Sharding(分片)的特性，图7示出根据本申请一优选的实施例提供的一种应用所述用于过滤数据设备的系统的设备示意图，多个设备1形成多个功能完全相同的对等节点进行集群，并使使集群具备了负载均衡能力，满足在海量数据背景下的可扩展性，高可用性和性能要求。
优选的，所述步骤S11还包括：将所述结构化待过滤数据发送至阻塞队列；相应地，所述步骤S13包括：从所述阻塞队列中获取所述结构化待过滤数据。
在此，所述阻塞队列能够在队列满的时候阻塞进一步的入队操作直至所述阻塞队列的队列不满。具体地，所述步骤S11将所述结构化待过滤数据发送至阻塞队列，则所述结构化待过滤数据进入阻塞队列等待，所述步骤S13根据所述结构化待过滤数据的等待次序，依此从所述阻塞队列中获取所述结构化待过滤数据，当所述结构化待过滤数据被获取后即从所述阻塞队列删除。在此，所述阻塞队列中等待的结构化待过滤数据占满所 述阻塞队列时，所述阻塞队列阻塞所述步骤S11发送过滤数据进入阻塞队列的操作，从而能够避免处理能力不足时，内存占用过大，从而在海量数据过滤过程中起到削峰填谷的作用，避免处理故障。
进一步地，在所述步骤S12中，加载过滤规则，其中，每一所述过滤规则包括：规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表。
在此，所述规则领域标识用于表明所述过滤规则的类别，其中，所述的类别例如且不限于：主机的CPU占用率、某网站的访问延迟时间等，所述规则领域标识可以采用数据或文字等进行标识，此外，任何能够被计算机识别的标识的方式都可以作为所述数据领域标识的实施方式，并以引用的方式包含于此。优选地，所述规则领域标识的内容与所述数据领域标识的内容相同或基本相同，以便所述步骤S13根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则。其中，所述规则名称可以是全局唯一识别的规则名称，以便于过滤规则的管理维护。其中，规则运算表达式可以是数字、字符串形式组成的规则表达式，例如(仅为举例，并不限于此)：instanceId＝'AY123456'||clusterId＝hangzhou)&&value>80，规则运算表达式还可以包括非数字、字符串等原生类型组成的数据集合类型，例如(仅为举例，并不限于此)：数组、哈希集合等。
进一步地，所述步骤S12包括：建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表，其中，所述第一规则列表用于为步骤S13获取过滤规则提供支持。
进一步地，在所述步骤S13中，获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则。具体地，所述步骤S13根据所述数据领域标识从所述第一规则列表中获取具有与数据领域标识相对应的规则领域标识的若干过滤规则。
进一步地，在所述步骤S14中，利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
优选的，对于每一所述结构化待过滤数据，所述步骤S13根据其数据领域标识获取若干具有相应相同的规则领域标识的若干过滤规则，则步骤S14利用每一所获取的过滤规则对所述结构化待过滤数据进行一次匹配运算，所述步骤S14对若干所获取的过滤规则进行并行匹配计算，以充分利用多核中央处理器的性能，提高过滤效率。
具体地，所述步骤S14包括：对所获取的过滤规则进行规则编译，以建立可运行抽 象语法树；将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算。
所述步骤S14实现了抽象语法树的功能，能够支持算术运算，字符串运算，关系运算，逻辑运算，正则表达式运算和集合运算等，并预留了扩展接口，可支持用户自定义的运算等。
进一步地，对所获取的过滤规则进行规则编译，以建立可运行抽象语法树(AST，Abstract Syntax Tree)，在此，所述抽象语法树是是规则表达式的抽象语法结构的树状表现形式。
其中，对所获取的过滤规则进行规则编译，以建立可运行抽象语法树包括：用于对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树，具体地，可以采用Antlr(Another Tool for Language Recognition)来实现，能够将用户自定义的过滤规则表达式转化为抽象语法树；通过对规则表达式进行词法分析得到AST的Token流，Token流(令牌)包括分析识别出的字符串规则各种运算操作符，运算操作符包括但不限于例如：操作符、数字、字符串、变量、正则表达式等。
其中，运算操作符的示例代码与图1所示设备1的第四装置14转化的抽象语法树的运算操作符的示例代码的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
在具体的应用场景中，例如过滤规则的规则表达式为以下字符串形式的内容：
CPU>90/100and clusterId in[‘hz’,’qd’]and instanceId like‘AK47\w+’
通过编写Antlr词法分析规则，得到如图9所示的AST token流，并在系统中的保存形式采用运算表达式后缀表示法来解决优先级问题，如图8所示，保存形式为：OP:操作符，Num:数字，Var:变量，Regex:正则表达式，StrArray:字符串数组。
接着，对所述抽象语法树进行预计算，以获得所述可运行抽象语法树。其中，所述预计算用于将AST token流中的常量表达式进行预先计算，以判断子表达式是否可计算，并通过预计算检查所述抽象语法树中每个元素是否为特殊类型，将其中的特殊类型的元素转换为程序语言数据结构元素，例如但不限于，将Like操作参数元素解释转化为正则表达式，将In操作参数元素解释转化为集合。通过预计算能够将AST中的常量表达式进行预先预算，从而加快运行时处理速度，并特殊类型的元素转化为程序语言数据结构元素，其中，所述特殊类型的元素是非数字、字符串组成的原生类型的元素，例如但不限于数据集合类型，例如但不限于数组、哈希map、哈希集合等。
接上例，在具体的场景中，对图9所示的所述抽象语法树进行一次预计算，计算结果为可运行抽象语法树(AST)，其中，AST的token流如图10所示，其中，“0.9”、“Java.util.HashSet[‘hz’，‘qd’]”及“Java.util.regex.pattern‘AK47\W+’”为经过预计算的计算结果。
进行预计算的示例代码可以与图1所示第四装置14进行预计算的示例代码的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
具体地，对所述抽象语法树进行预计算包括：
根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中；当所述元素为操作符时，将操作符对应的两个操作数传出运行堆栈，计算以获得计算结果；当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入运行堆栈中。
进一步地，将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的过程，利用若干所述可运行抽象语法树进行并行匹配计算运算的过程与一趟预计算相同，在正常情况下运行时AST中的所有表达式均为可计算表达式，所以最终的计算结果是一个确定的值，该值为布尔值FALSE或者TRUE，如果计算结果的布尔值为TRUE，则该结构化数据则判断为满足该过滤规则。
在此，所述利用若干所述可运行抽象语法树进行并行匹配计算通过利用所述可运行抽象语法树对所述数据进行匹配计算，例如，当本申请所述设备1被分配到了1000个过滤规则，则对于每一所述结构化待过滤数据，在所述设备1的线程池中对这1000个过滤规则并发执行匹配运算，以充分利用多核CPU的性能来并发计算过滤规则。
具体地，利用若干所述可运行抽象语法树进行并行匹配计算包括：将所述可运行抽象语法树的变量替换为所述数据体中的参数；利用所述运行堆栈对所述可运行抽象语法树进行匹配计算。
将所述可运行抽象语法树的变量替换为所述数据体中的参数的示例代码与所述图1中的设备1的第四装置14替换示例代码的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
对AST中的操作符节点进行相应的运算的示例代码与所述图1中的设备1的第四装置14进行相应的运算的示例代码的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
同样地，进行匹配计算与所述图1中的设备1的第四装置14进行匹配计算的示例代码的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
此后，所述方法还可以对该结构化待过滤数据进行进一步处理，例如报警等。
图5示出根据本申请一优选实施例提供的一种用于过滤数据的方法流程示意图，所述方法包括：步骤S11’、步骤S12’、步骤S13’、步骤S14’和步骤S15’。
所述步骤S11’、步骤S13’和步骤S14’的内容与图4所示的步骤S11、步骤S12和步骤S14的内容相同或基本相同，为简明起见，不再赘述，仅以引用的方式包含于此。
优选的，所述步骤S12’在引用图4所示的步骤S12的内容的基础上，所述步骤S12’还包括：建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表；所述步骤S12’按照所述过滤规则的规则领域标识和所述过滤规则的规则名称为两个维度的索引建立第一规则列表和第二规则列表，其中，以所述过滤规则的规则领域标识为索引的第一规则列表供过滤数据时查找，以所述过滤规则的规则名称为索引的第二规则列表供过滤规则的管理和维护时查找。获得所述结构化待过滤数据时，根据数据领域标识匹配查找第一规则列表中的过滤规则，找到对应的过滤规则的列表，并遍历该过滤规则的列表，将格式化待过滤数据的数据体作为输入参数，对列表中的每一规则做并发匹配计算。所述第二规则列表便于对过滤规则进行管理。
在所述步骤S15’中，新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
具体地，所述步骤S15’包括以下至少任一项：将新增的过滤规则增加至所述第二规则列表中；从所述第二规则列表中删除相应的过滤规则；从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译，所述步骤S15’能够对过滤规则进行修改和增删操作，提高过滤规则的灵活性。
图6示出根据本申请另一优选实施例提供的一种用于过滤数据的方法流程图，其中，所述方法包括步骤S11”、步骤S12”、步骤S13”、步骤S14”、步骤S15”和步骤S16”。
其中，所述步骤S11”、步骤S12”、步骤S13”、步骤S14”和步骤S15”与图5所示的步骤S11’、步骤S12’、步骤S13’、步骤S14’和步骤S15”的内容相同或基本相同，为简明起见，不再赘述，并以引用的方式包含于此。
在此，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；在所述步骤S16’中，将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。在此，所述通知器是对预留通知接口的一组实现，可实现自定义的通 知方式，比如使用不同的传输协议，不同的压缩算法，不同的序列化算法传输至下游系统集群中不同的系统中。其中，所述通知器可以在过滤规则创建时，即进行自由地组合装配绑定至任意过滤规则。
与现有技术相比，根据本申请一实施例所提供的用于数据过滤的设备及方法采用流式运算方式，不会在内存中缓存也不会固化数据，即每次获取初始待过滤数据后将所述初始待过滤数据转换为结构化待过滤数据，并利用相应的过滤规则实时进行匹配计算，随即得到过滤结果，解决海量流式数据的过滤的实时性问题；
进一步地，根据本申请一实施例所提供的用于数据过滤的设备及方法所述过滤数据的方法和设备支持算术运算，字符串运算，关系运算，逻辑运算，正则表达式运算，集合运算，且预留了扩展接口，并且所述过滤规则为简单的带变量的运算表达式形式，解决了过滤规则描述复杂、扩展不易及管理困难的问题；
此外，本申请根据本申请一实施例所提供的用于数据过滤的设备及方法为自主设计开发，成本相对较低，且在任意代码路径上都可以监控和调优。
经多次性能测试，得到的性能指标大致为在单台4核8G配置的虚拟机可以支撑50万过滤规则，处理流式数据TPS达到20000，过滤出有效数据TPS达到2000，系统平均负载稳定在load1-4左右，CPU资源得到了有效利用。
在本申请一个典型的配置中，终端、服务网络的设备和可信方均包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括非暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
需要注意的是，本申请可在软件和/或软件与硬件的组合体中被实施，例如，可采用 专用集成电路(ASIC)、通用目的计算机或任何其他类似硬件设备来实现。在一个实施例中，本申请的软件程序可以通过处理器执行以实现上文所述步骤或功能。同样地，本申请的软件程序(包括相关的数据结构)可以被存储到计算机可读记录介质中，例如，RAM存储器，磁或光驱动器或软磁盘及类似设备。另外，本申请的一些步骤或功能可采用硬件来实现，例如，作为与处理器配合从而执行各个步骤或功能的电路。
另外，本申请的一部分可被应用为计算机程序产品，例如计算机程序指令，当其被计算机执行时，通过该计算机的操作，可以调用或提供根据本申请的方法和/或技术方案。而调用本申请的方法的程序指令，可能被存储在固定的或可移动的记录介质中，和/或通过广播或其他信号承载媒体中的数据流而被传输，和/或被存储在根据所述程序指令运行的计算机设备的工作存储器中。在此，根据本申请的一个实施例包括一个装置，该装置包括用于存储计算机程序指令的存储器和用于执行程序指令的处理器，其中，当该计算机程序指令被该处理器执行时，触发该装置运行基于前述根据本申请的多个实施例的方法和/或技术方案。
对于本领域技术人员而言，显然本申请不限于上述示范性实施例的细节，而且在不背离本申请的精神或基本特征的情况下，能够以其他的具体形式实现本申请。因此，无论从哪一点来看，均应将实施例看作是示范性的，而且是非限制性的，本申请的范围由所附权利要求而不是上述说明限定，因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本申请内。不应将权利要求中的任何附图标记视为限制所涉及的权利要求。此外，显然“包括”一词不排除其他单元或步骤，单数不排除复数。装置权利要求中陈述的多个单元或装置也可以由一个单元或装置通过软件或者硬件来实现。第一，第二等词语用来表示名称，而并不表示任何特定的顺序。

Claims (18)

  1. 一种用于过滤数据的方法，其中，所述方法包括：
    获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；
    加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；
    获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；
    利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算。
  2. 根据权利要求1所述的方法，其中，获取初始待过滤数据包括：
    从分布式消息中间件中获取所述初始待过滤数据。
  3. 根据权利要求1或2所述的方法，其中，将所述初始待过滤数据转换为结构化待过滤数据还包括：
    将所述结构化待过滤数据发送至阻塞队列；
    获取所述结构化待过滤数据包括：
    从所述阻塞队列中获取所述结构化待过滤数据。
  4. 根据权利要求1至3中任一项所述的方法，其中，利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配运算包括：
    对所获取的过滤规则进行规则编译，以建立可运行抽象语法树；
    将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算。
  5. 根据权利要求4所述的方法，其中，对所获取的过滤规则进行规则编译，以建立可运行抽象语法树包括：
    对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树；
    对所述抽象语法树进行预计算，以获得所述可运行抽象语法树；
    其中，对所述抽象语法树进行一次预计算包括：
    根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中；
    当所述元素为操作符时，将所述操作符对应的两个操作数传出所述运行堆栈，计算以获得计算结果；
    当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入所述运行堆栈中。
  6. 根据权利要求4或5所述的方法，其中，利用若干所述可运行抽象语法树进行并行匹配计算包括：
    将所述可运行抽象语法树的变量替换为所述数据体中的参数；
    利用所述运行堆栈对所述可运行抽象语法树进行匹配计算。
  7. 根据权利要求1至6中任一项所述的方法，其中，所述方法还包括：
    新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
  8. 根据权利要求7所述的方法，其中，建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表还包括：
    建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表；
    所述新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译包括至少以下任一项：
    将新增的过滤规则增加至所述第二规则列表中；
    从所述第二规则列表中删除相应的过滤规则；
    从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译。
  9. 根据权利要求1至8中任一项所述的方法，其中，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；
    所述方法还包括：
    将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。
  10. 一种用于过滤数据的设备，其中，所述设备包括：
    第一装置，用于获取初始待过滤数据，并将所述初始待过滤数据转换为结构化待过滤数据，其中，所述结构化待过滤数据包括数据领域标识和键-值对格式的数据体；
    第二装置，用于加载过滤规则，其中，每一所述过滤规则包括规则领域标识、规则名称及规则运算表达式，并建立以所述过滤规则的领域标识为索引的所述过滤规则的第一规则列表；
    第三装置，用于获取所述结构化待过滤数据，并根据所述数据领域标识从所述第一规则列表中获取具有与所述数据领域标识相对应的规则领域标识的若干过滤规则；
    第四装置，用于利用所获取的若干过滤规则对所述结构化待过滤数据进行并行匹配 运算。
  11. 根据权利要求10所述的设备，其中，所述第一装置包括：
    从分布式消息中间件中获取所述初始待过滤数据的单元。
  12. 根据权利要求10或11所述的设备，其中，所述第一装置包括：
    用于将所述结构化待过滤数据发送至阻塞队列的单元；
    所述第三装置包括：
    从所述阻塞队列中获取所述结构化待过滤数据的单元。
  13. 根据权利要求10至12中任一项所述的设备，其中，所述第四装置包括：
    用于对所获取的过滤规则进行规则编译，以建立可运行抽象语法树的单元；
    用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元。
  14. 根据权利要求13所述的设备，其中，所述用于对所获取的过滤规则进行规则编译，以建立可运行抽象语法树的单元包括：
    用于对所获取的过滤规则的规则表达式进行分析，以转化为抽象语法树的模块；
    用于对所述抽象语法树进行预计算，以获得所述可运行抽象语法树的模块，其中，该模块用于：
    根据所述抽象语法树创建运行堆栈，将所述抽象语法树中的元素传入所述运行堆栈中，
    当所述元素为操作符时，将操作符对应的两个操作数传出所述运行堆栈，计算以获得计算结果，
    用于当所述元素为特殊元素，则将所述特殊元素转换为程序语言数据结构元素后传入所述运行堆栈中。
  15. 根据权利要求13或14所述的设备，其中，所述用于将所述结构化待过滤数据的数据体作为输入参数，遍历若干所述可运行抽象语法树，并利用若干所述可运行抽象语法树进行并行匹配计算的单元包括：
    用于将所述可运行抽象语法树的变量替换为所述数据体中的参数的模块；
    用于利用所述运行堆栈对所述可运行抽象语法树进行匹配计算的模块。
  16. 根据权利要求10至15中任一项所述的设备，其中，所述设备还包括：
    第五装置，用于新增过滤规则、删除过滤规则或对已有的过滤规则进行修改编译。
  17. 根据权利要求16所述的设备，其中，所述第二装置还包括：
    建立根据所述过滤规则的规则名称为索引的所述过滤规则的第二规则列表的单元；
    所述第五装置包括：
    用于将新增的过滤规则增加至所述第二规则列表中的单元；
    用于从所述第二规则列表中删除相应的过滤规则的单元；
    用于从所述第二规则列表中查找过滤规则，并对所查找的过滤规则进行修改编译的单元。
  18. 根据权利要求10至17中任一项所述的设备，其中，每一所述过滤规则还包括：所述过滤规则所绑定的通知器的信息；
    所述设备还包括：
    第六装置，用于将满足相应所述过滤规则的结构化待过滤数据发送至该过滤规则所绑定的通知器，以备传输。
PCT/CN2016/088302 2015-07-13 2016-07-04 一种用于过滤数据的设备及方法 Ceased WO2017008650A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510408180.1 2015-07-13
CN201510408180.1A CN107038161B (zh) 2015-07-13 2015-07-13 一种用于过滤数据的设备及方法

Publications (1)

Publication Number Publication Date
WO2017008650A1 true WO2017008650A1 (zh) 2017-01-19

Family

ID=57757755

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/088302 Ceased WO2017008650A1 (zh) 2015-07-13 2016-07-04 一种用于过滤数据的设备及方法

Country Status (2)

Country Link
CN (1) CN107038161B (zh)
WO (1) WO2017008650A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110287174A (zh) * 2019-05-09 2019-09-27 北京善义善美科技有限公司 一种数据过滤引擎及系统和过滤方法
CN112565338A (zh) * 2020-11-10 2021-03-26 中国人民解放军战略支援部队信息工程大学 一种以太网报文捕获、过滤、存储、实时解析方法及系统
CN115047835A (zh) * 2022-06-27 2022-09-13 中国核动力研究设计院 基于dcs系统定期试验数据获取方法、装置、设备及介质
CN116383290A (zh) * 2023-03-22 2023-07-04 中国华能集团有限公司北京招标分公司 一种数据泛化解析方法

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107766466A (zh) * 2017-09-29 2018-03-06 上海望友信息科技有限公司 数据类型的识别方法、系统、计算机可读存储介质及设备
CN109672704B (zh) * 2017-10-16 2022-02-25 阿里巴巴集团控股有限公司 消息的处理方法、装置及电子设备
CN107766538A (zh) * 2017-10-28 2018-03-06 杭州安恒信息技术有限公司 基于java的数据过滤处理模块及同步、异步过滤方法
CN109189807A (zh) * 2018-09-13 2019-01-11 北京奇虎科技有限公司 一种报警数据的过滤方法和装置
CN110427754B (zh) * 2019-08-12 2024-02-13 腾讯科技（深圳）有限公司 网络应用攻击检测方法、装置、设备及存储介质
CN111427915B (zh) * 2020-03-25 2024-06-14 京东科技控股股份有限公司 信息处理方法及装置、存储介质、电子设备
CN112068933B (zh) * 2020-09-02 2021-08-10 成都鱼泡科技有限公司 一种实时分布式数据监听方法
CN114611500B (zh) * 2022-03-08 2025-10-28 湖南亚信安慧科技有限公司 表达式处理方法、装置、电子设备及计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102467561A (zh) * 2010-11-19 2012-05-23 金蝶软件（中国）有限公司 表格数据过滤方法及装置
CN103034700A (zh) * 2012-12-05 2013-04-10 北京奇虎科技有限公司 富文本内容的处理方法及系统
CN103618733A (zh) * 2013-12-06 2014-03-05 北京中创信测科技股份有限公司 一种应用于移动互联网的数据过滤系统及方法
CN104317947A (zh) * 2014-11-07 2015-01-28 烽火通信科技股份有限公司 一种基于海量数据的实时结构化数据比对系统
US8949371B1 (en) * 2011-09-29 2015-02-03 Symantec Corporation Time and space efficient method and system for detecting structured data in free text

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100508453C (zh) * 2006-09-19 2009-07-01 清华大学 一种开放式真实IPv6源地址过滤与验证方法
CN100485684C (zh) * 2006-10-08 2009-05-06 中国科学院软件研究所 文本内容过滤方法和系统
CN101127774B (zh) * 2007-09-19 2012-03-28 中兴通讯股份有限公司 初始过滤规则的优先级处理方法
CN101304589A (zh) * 2008-04-14 2008-11-12 中国联合通信有限公司 利用短信网关发送垃圾短信的监控与过滤方法及系统
CN101282332B (zh) * 2008-05-22 2011-05-11 上海交通大学 面向网络安全告警关联的攻击图生成系统
CN101414929B (zh) * 2008-11-18 2011-09-14 华为技术有限公司 一种信息获取的方法、装置及系统
CN101860531B (zh) * 2010-04-21 2013-04-17 北京星网锐捷网络技术有限公司 数据包过滤规则匹配方法及装置
CN102082728B (zh) * 2010-12-28 2012-07-25 北京锐安科技有限公司 一种网络审计系统的过滤规则动态加载方法
CN102654864A (zh) * 2011-03-02 2012-09-05 华北计算机系统工程研究所 一种面向实时数据库的独立透明型安全审计保护的方法
CN102231134A (zh) * 2011-07-29 2011-11-02 哈尔滨工业大学 基于静态分析的冗余代码缺陷检测方法
CN103116620B (zh) * 2013-01-29 2016-01-20 国家电网公司 基于策略的非结构化数据安全过滤方法
US9197632B2 (en) * 2013-03-15 2015-11-24 Kaarya Llc System and method for account access
CN103338155B (zh) * 2013-07-01 2016-02-24 安徽中新软件有限公司 一种数据包的高效过滤方法
CN103631966B (zh) * 2013-12-18 2017-10-10 用友网络科技股份有限公司 一种可配置的解析多值匹配字段的方法
CN103780460B (zh) * 2014-01-15 2017-06-30 珠海市佳讯实业有限公司 一种通过fpga实现tap设备硬件过滤的系统
CN104331278B (zh) * 2014-10-15 2017-08-25 南京航空航天大学 一种用于arinc661规范的指令过滤方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102467561A (zh) * 2010-11-19 2012-05-23 金蝶软件（中国）有限公司 表格数据过滤方法及装置
US8949371B1 (en) * 2011-09-29 2015-02-03 Symantec Corporation Time and space efficient method and system for detecting structured data in free text
CN103034700A (zh) * 2012-12-05 2013-04-10 北京奇虎科技有限公司 富文本内容的处理方法及系统
CN103618733A (zh) * 2013-12-06 2014-03-05 北京中创信测科技股份有限公司 一种应用于移动互联网的数据过滤系统及方法
CN104317947A (zh) * 2014-11-07 2015-01-28 烽火通信科技股份有限公司 一种基于海量数据的实时结构化数据比对系统

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110287174A (zh) * 2019-05-09 2019-09-27 北京善义善美科技有限公司 一种数据过滤引擎及系统和过滤方法
CN112565338A (zh) * 2020-11-10 2021-03-26 中国人民解放军战略支援部队信息工程大学 一种以太网报文捕获、过滤、存储、实时解析方法及系统
CN115047835A (zh) * 2022-06-27 2022-09-13 中国核动力研究设计院 基于dcs系统定期试验数据获取方法、装置、设备及介质
CN115047835B (zh) * 2022-06-27 2024-06-04 中国核动力研究设计院 基于dcs系统定期试验数据获取方法、装置、设备及介质
CN116383290A (zh) * 2023-03-22 2023-07-04 中国华能集团有限公司北京招标分公司 一种数据泛化解析方法
CN116383290B (zh) * 2023-03-22 2023-10-31 中国华能集团有限公司北京招标分公司 一种数据泛化解析方法

Also Published As

Publication number Publication date
CN107038161A (zh) 2017-08-11
CN107038161B (zh) 2021-03-26

Similar Documents

Publication Publication Date Title
CN107038161B (zh) 一种用于过滤数据的设备及方法
US11537371B2 (en) System and method for metadata-driven external interface generation of application programming interfaces
US11216302B2 (en) Modifying task dependencies at worker nodes using precompiled libraries
US10592282B2 (en) Providing strong ordering in multi-stage streaming processing
US10198298B2 (en) Handling multiple task sequences in a stream processing framework
US9563663B2 (en) Fast path evaluation of Boolean predicates
Mehmood et al. Distributed real-time ETL architecture for unstructured big data
US9015080B2 (en) Systems and methods for semantic inference and reasoning
US11657069B1 (en) Dynamic compilation of machine learning models based on hardware configurations
CN110431545A (zh) 针对结构化数据和非结构化数据执行查询
US11636124B1 (en) Integrating query optimization with machine learning model prediction
US10417058B1 (en) Method and system for executing application programming interface (API) requests based on parent-child object relationships
CN106104533A (zh) 处理大型数据储存库中的数据集
Bagui et al. Positive and negative association rule mining in Hadoop’s MapReduce environment
US20180129712A1 (en) Data provenance and data pedigree tracking
US10108745B2 (en) Query processing for XML data using big data technology
US20250156292A1 (en) Duplication Avoidance For Mutable Event Based Metrics
Venkatesan et al. PoN: Open source solution for real-time data analysis
US12547925B1 (en) Adapting concurrency to support machine learning model prediction
Ventocilla Big data programming with Apache spark
Mishra Titan graph databases with cassandra
Singhal et al. Comparative analysis of big data technologies
Zannelli Data Quality for streaming applications

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16823797

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16823797

Country of ref document: EP

Kind code of ref document: A1