Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2455—Query execution
  • G06F16/24568—Data stream processing; Continuous queries
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2455—Query execution
  • G06F16/24564—Applying rules; Deductive queries
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2457—Query processing with adaptation to user needs
  • G06F16/24575—Query processing with adaptation to user needs using context
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/25—Integrating or interfacing systems involving database management systems
  • G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/54—Interprogram communication
  • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N5/00—Computing arrangements using knowledge-based models
  • G06N5/04—Inference or reasoning models

Definitions

• CEP complex event processing
• Figures 5A-5D are diagrams illustrating template examples according to one embodiment.
• FIGS. 6A and 6B illustrate a representation of a template in JavaScript Object Notation (JSON) according to one embodiment.
• JSON JavaScript Object Notation
• Figure 12 is a flow diagram illustrating a method to process constraints according to another embodiment.
• Figure 14 is a flow diagram illustrating another method of event retention according to one embodiment.
• Figure 15 is a block diagram illustrating a data processing system according to one embodiment.
• a template type refers to the specification of a set of assertions and constraints that should be monitored by the system.
• a template instance of solution refers to a set of events that collectively satisfy a template. This is also called a solution to the template. For example, one event can participate in multiple solutions. Each solution will satisfy all constraints in the template. It may not satisfy all multiplicity requirements in which case the actions from those assertions will not be implied.
• An assertion refers to a part of a template definition indicating some sets of data that if true validate the hypothesis represented by a template definition. Most commonly this will be reference to an event type indicating that events of that type can potentially satisfy the assertion if all constraints are met.
• An outcome refers to an action to be initiated by the system when the assertions the action depends upon are all true. This may be an action to publish an event.
• a group refers to a set of assertions that are collectively satisfied for the group to be satisfied.
• a constraint refers to a specification of restrictions on the events considered to be satisfying the assertions to which the constraint applies. For example, a constraint that withdraws and deposits must be from the same account is a constraint on withdrawal and deposit events/assertions.
• a field refers to a piece of data in an event used as input to a constraint or action.
• An event type (or event definition) refers to a specification for generation of events from ingested data, or to be published from templates. The specification includes definition of fields to be included in the processed events.
• input source data that includes event data of one or more events is received.
• One or more event definitions that match the event data are selected.
• the event definition is inputted into a template to generate a set of events.
• the template includes a number of assertions and has the event definition as one of the assertions, where each assertion includes a constraint.
• the constraints of the assertions are progressively processed to produce one or more solutions that are subsets of the set of events. For each constraint and each solution, a set of target events that is viable for the solution is identified, and a new solution is produced based on the solution and the identified set of target events, whereby a set of new solutions is produced.
• each new solution and each outcome in the template it is determined whether all of the assertions are satisfied. In response to determining that all of the assertions are satisfied, it is determined whether an equivalent outcome has previously triggered. The outcome is triggered in response to determining that the equivalent outcome has not previously triggered.
• the outcome in response to determining that at least one assertion is not satisfied and the equivalent outcome has previously triggered, the outcome is un-triggered. [0028] In one embodiment, to trigger the outcome, event data of the new solution is created, and the event data for the new solution is published.
• to publish the event data of the new solution basic data for the new solution is generated. For each data mapping entry in the outcome, source event values from source events referenced in the data mapping entry are collected, a computation on the source event values is performed to produce a result, and the result is placed in the new solution. Entity data is extracted from the source events. The entity data is aggregated into an entity map, wherein the entity map identifies each entity referenced in the source events.
• non-partitioning constraints are grouped into stages.
• the stages are ordered to form a hierarchical data structure.
• Each event is placed into one of a number of partial solutions in each of the assertions having an event definition of the event, where the partial solutions are included in the hierarchical data structure.
• the partial solutions are separated based on a partition key. Partial solutions having a common partition key are combined to form one or more complete solutions.
• the combined partial solutions are processed for outcome.
• Event data comprising passive event data, active event data, or both is received. It is determined whether the received event data is available for a pattern of passive event data and active event data. In response to determining that the received event data is available for the pattern of passive event data and active event data, one or more constraints between the passive event data and the active event data are converted into one or more query terms. The query terms are used to construct at least one query. Remaining passive event data that is related to some, but not all, of the active event data is obtained using the constructed at least one query. One or more event definitions that match the event data and the remaining passive event data are selected.
• User devices 101-102 may provide an electronic display along with an input/output capability.
• a separate electronic display and input/output device e.g., keyboard
• server 150 Any of a wide variety of electronic displays and input/output devices may be utilized with system 100.
• a user may utilize user devices 101-102 to access a web-page hosted by the server 150 through the network 103.
• Server 150 may provide a webpage the user can access by using a conventional web browser or viewer, e.g., Safari, Internet Explorer, and so forth, that can be installed on user devices 101-102.
• Published events 165 (as described in more detail herein below) may be presented to the user through the webpage.
• server 150 may provide a computer application that can be downloaded to user devices 101-102.
• the user can access a web-page hosted by server 150 to download the computer application.
• the computer application can be installed on user devices 101-102, which provides the interface for the user to view the published events.
• server 150 is communicatively coupled or connected to external system 171 and data storage unit 172, which may be over a network similar to network 103.
• Server 150 may be any kind of server or a cluster of servers, such as Web or cloud servers, application servers, backend servers, or a combination thereof.
• server 150 may rely on one or more external sources of input events and produce one or more sets of output (also events).
• the sets of output can act as an input to server 150 itself.
• server 150 selects events of interest, compares them to defined templates, filters them for combinations of events that match template specific constraints, and potentially publishes new events for matching templates, which can in turn be matched by templates. The results can be used to detect patterns of behavior represented by the matched events, generate alerts for combinations of events of interest, and synthesize summary information from low level pieces of data.
• External system 171 can be any computer system with computational and network-connectivity capabilities to interface with server 150.
• external system 171 may include multiple computer systems. That is, external system 171 may be a cluster of machines sharing the computation and source data storage workload.
• data storage unit 172 may be any memory storage medium, computer memory, database, or database server suitable to store electronic data. Data storage unit 172 may be a separate computer independent from server 150. Data storage unit 172 may also be a relational data storage unit. In one embodiment, data storage unit 172 may reside on external system 171, or alternatively, can be configured to reside separately on one or more locations.
• server 150 includes, but not limited to, input data receiving module 151, event processing module 152, solution generation module 153, outcome processing module 154 and event publishing module 155.
• modules 151-155 may be implemented in software, hardware, or a combination thereof.
• these modules may be installed on persistent storage device 182 (e.g., hard disk, solid state drive), loaded into memory 181, and executed by one or more processors (not shown) of server 150.
• persistent storage device 182 e.g., hard disk, solid state drive
• processors not shown
• server 150 includes, but not limited to, input data receiving module 151, event processing module 152, solution generation module 153, outcome processing module 154 and event publishing module 155.
• persistent storage device 182 e.g., hard disk, solid state drive
• processors not shown
• streaming/active (or push) data sources can be combined with passive (or pull) data sources to improve the accuracy of the events obtained and the timeliness of the events obtained from passive data stores (e.g., structured query language (SQL) databases, web services, etc.).
• passive data stores e.g., structured query language (SQL) databases, web services, etc.
• the query term utilizes them with appropriate constraints to generate an optimal query. For example, if an active event contains a time value and the passive events are constrained to be +/- 5 minutes from an active event time value, a query can be constructed to identify the events in the passive data source that are within that time window. When the number of events is limited in the passive event set the query is constructed to obtain the events closest to a targeted value (time, for example). This may require that two queries be performed to obtain values above/below the target value. Once a set of events for a passive assertion have been obtained they are streamed into the pattern as if they were active, and this may trigger more constraint processing, possibly including more passive data sources.
• a template instance with matching events is determined by module 153, it is evaluated for actions, such as actions 164, to take by outcome processing module 154.
• the actions 164 may be predetermined and stored on storage device 182. If the multiplicity of all assertions is met for input to an action, module 154 may trigger that action.
• a triggered action can access the input events that initiated the action. In the case where a previously triggered action is not triggered after the solution being updated it is untriggered. Thus, if a solution matches the set of multiplicities and a new event arrives, that means it no longer does so, the previously triggered action is un-triggered. In the case of a published event that event is retracted when un-triggered. This causes the event to be removed from all solutions that may include it and those template instances re-evaluated.
• FIG. 2 is a flow diagram illustrating a method of processing an event according to one embodiment.
• Process 200 may be performed by processing logic which may include software, hardware, or a combination thereof.
• processing logic may include software, hardware, or a combination thereof.
• server 150 e.g., some or all of modules 151-155.
• the processing logic receives input source data that includes event data of one or more events.
• the processing logic selects one or more event definitions that match the event data.
• the processing logic inputs the event definition into a template to generate a set of events.
• the template includes a number of assertions and has the event definition as one of the assertions, where each assertion includes a constraint.
• the processing logic progressively processes the constraints of the assertions to produce one or more solutions that are subsets of the set of events.
• the processing logic identifies a set of target events that is viable for the solution, and produces a new solution based on the solution and the identified set of target events, whereby a set of new solutions is produced.
• the processing logic forwards the data to the relevant template type processing elements (1103) that have that event type as an assertion in their template definition.
• the processing logic updates the set of solutions for the known events. This update can occur immediately, after a delay, or upon receipt of sufficient number of new events based on latency and throughput requirements.
• a solution is a subset of all events sent to the template processing element (also referred to as module, unit, or logic) that collectively match assertions of the template definition and satisfy all constraints between those assertions.
• the number of required events in the assertion is a secondary condition on whether the solution “matches” and does not limit the events being considered in the solution.
• any affected template processing elements persist, in operation 310, the processing logic updates their state to record or track all received and published events. It should be noted that the state for each solution may be tracked and persisted independently of other solutions, that state updates may occur continuously during the computation process or at the end, and that state updates may occur while processing one event or a small number of events to meet throughput or latency requirements.
• Source events can have a multiplicity specification, which indicates the minimum and maximum number of such events are to be matched to one template instance (one set of conforming/matching events). For example, a template looking for money transfers would look for one withdraw followed by a deposit, while a different template may look for at least 3 people from a specific group who book travel to the same city.
• aspects of the disclosure include the publishing of events to represent a condition hypothesized by a template.
• a published event may be a type of action or outcome of a template and represented, for example by a rounded comer rectangle with a text name label as shown herein below, and is connected to each source event or group. Other outcomes may include alerts or invocation of specific programming logic.
• Each source event is connected to the outcome, for example by a line.
• each outcome to publish an event may include a map defining data mapping from the source events to the published event.
• a time range of all source events is automatically recorded in the published event, as is the source template instance.
• the system may record in the published event minimum and/or maximum times of all input events.
• the template author can indicate that select fields from the source events are to be propagated to the published event.
• the event is published as soon as an event is matched to each connected source event, even if there are more source events on the template.
• this allows a partial match of a template to still publish events when warranted.
• a template may wish to publish an event when a critical phase of a process has been detected even if there are further steps in the process that could be detected.
• a primary constraint in templates is control of event time relationships.
• Temporal constraints allow an author to easily indicate that one event should follow another event within a time period, or to be within a certain tolerance of each other within a time period. Such constraints may apply to the time of the real world event, not to the time the data is processed or recorded by the system.
• Temporal constraints can include fuzzy match parameters such as A must follow B by 15 to 20 minutes, or A must follow B by 10 +/- 2 minutes.
• temporal constraints can require the difference in time be greater or less than a specified duration (less than 4 min, or at least 5 min, etc.).
• Temporal constraints may be represented as a directed arrow with any time constraints as a textual label over the line. Events can have more than one temporal value (such as air ticket booking time, departure time, and arrival time), in which case the temporal field name will be shown in the constraint text as in “arrival ⁇ departure” or “booking + 24h ⁇ departure”.
• a partition constraint separates the set of potentially matching events into subsets based on a field value.
• the field value may be an identifier for people, places, or things.
• the field value may be a quantity value such as a dollar amount or other measure being compared. This is used to ensure that events with different field values are not combined in the same template solution. For example when considering if travel to a restricted country matches the restrictions a partition constraint will be used to ensure that each solution considers only one person’s travel.
• each group acts like an “and” condition, while multiple input lines act as an “or” condition.
• Groups are represented, for example, as an ellipse enclosing the source events (e.g., fuel oil, fertilizer, detonator, etc.), as shown in Figure 4G.
• Figures 5A-5D are diagrams illustrating template examples according to one embodiment.
• the sample templates are provided to show the notation in context.
• the sample template e.g., Template 1
• the sample template e.g., Template 2
• the sample template e.g., Template 3
• the sample template e.g., Template 3
• the sample template e.g., Template 4
• the sample template may be the use of And/Or for more complex conditions.
• Figures 6A and 6B illustrate a representation of a template in JavaScript Object Notation (JSON) according to one embodiment.
• Figures 6A and 6B document the representation of one of the prior examples as JSON as one persistent representation of the notation.
• the notation may be represented as XML or RDF as well.
• aspects of the disclosure include template matching. For example, given a set E of events that meet one of the assertions of a template definition, and given C the set of all constraints on those assertions, the template matching component produces all subsets of E that satisfy the set of constraints C for which there are events in the target assertion of each of C. Note that in cases where an assertion is optional, events may or may not be present, and if not present, the corresponding constraints do not need to be met for a valid solution. Thus, solutions can have a subset of all defined assertions and still be considered a solution if they satisfy all constraints that have any of those assertions as a target. Each such solution results in a template instance in the system. Each instance is persisted and tracked as new events are received, and may be deleted if no longer applicable to the set of known events (most common when critical events are retracted as described in more detail herein below).
• a method of producing subsets of E is as follows:
• the processing logic sorts the constraints.
• partition constraints may be ordered such that each constraint is preceded by any that has constraints targeting the source of the partition constraint and source from the partition constraint. In other words, treating all constraints (source -> target) as a directed graph order partition constraints based on that directional ordering.
• the processing logic starts with a solution with all events, then progressively processing constraints to produce solutions that are subsets of the original set of events. That is, at operations 703-705, for each constraint the processing logic augments each solution produced by a prior constraint with matching events for that constraint. That is, the processing logic collects a set of target events that independently satisfy the constraint.
• This process performs a breadth first traversal (or search) of all possible solutions by progressively pruning the events in any one solution and producing new solutions where there are alternate viable solutions.
• the production of target events that are viable for the input solution is constraint specific logic.
• a new solution is produced from the input solution and the identified target events.
• the processing logic replaces the input solution with the new solution for each set of target events.
• multiple solutions can be produced from one input solution when multiple sets of target events are identified.
• the processing logic determines whether there are more solutions to update. If so, the processing logic proceeds to operation 704. Otherwise, the processing logic proceeds to operation 708 where the processing logic determines whether there are more constraints. If so, the processing logic proceeds to block 703. Otherwise, the processing logic proceeds to operation 709 where a set of viable solutions (e.g., solutions 163 of Figure IB) is produced for atemplate.
• a set of viable solutions e.g., solutions 163 of Figure IB
• Figure 8 is a flow diagram illustrating a method to generate solutions according to another embodiment.
• Process 800 may be performed by processing logic which may include software, hardware, or a combination thereof.
• processing logic may include software, hardware, or a combination thereof.
• process 800 may be performed by server 150, e.g., solution generation module 153.
• the processing logic processes events continuously as they arrive and fed to the constraint processing pipeline established by the ordering and grouping into stages.
• the processing logic may place each event received into a partial solution in each of the assertions having that event type. This allows constraints to work with partial solutions as input and output. Each constraint retains for each observed value in the constrained fields a list of matching partial solutions for each of its inputs when it is the first constraint of a stage. This allows it to produce new partial solutions combining both inputs upon receipt of any new input. For example: A constraint constrains assertion A with field identifier (id) to equal assertion B with field person id.
• Partition constraints ensure that events with different values in a selected field are not in the same solution. This partitions the events into distinct subsets. This is commonly used to ensure all events relate to the same person, location, organization, or activity. For example, a template looking for suspicious travel would look at travel records for a single person, while one looking at maintenance patterns would look at records for a single airline or facility. This constraint operates on equality described in the following paragraph.
• process 1100 in some embodiments strict equality may be used on string, identifier, or other unique values.
• the processing logic groups values in the set of target events by constrained value.
• the processing logic iterates each solution and uses source values to collect target events by source event values.
• the processing logic combines the source solution with target events for a new solution. That is, each distinct value in the source set results in the new solution output by the constraint.
• process 1100 is also used for partition constraints.
• Figure 12 is a flow diagram illustrating a method to process constraints (e.g., relational constraints) according to another embodiment.
• Process 1200 may be performed by processing logic which may include software, hardware, or a combination thereof.
• process 1200 may be performed by server 150, e.g., solution generation module 154.
• FIG. 13 is a flow diagram illustrating a method of event retention according to one embodiment.
• Process 1300 may be performed by processing logic which may include software, hardware, or a combination thereof.
• processing logic may include software, hardware, or a combination thereof.
• server 150 e.g., input data processing module 151.
• the query results can be retained to allow modification of the constraint resolution when active events are revised.
• the passive source cannot be re-queried to process the modification because the passive source data may have changed thereby yielding inconsistent results. For example, if an active event is deleted from the data stream, then any passive data related to that event may need to be deleted from the set of events for this pattern and may require a revised output for the pattern as a whole. The same for modification of a value used in constructing a query that may return different events and thus some of the prior events may need to be deleted to adjust for the new value.
• a query can be constructed to use all other constraints to obtain the set of A and B that meet all other constraints, and then narrow it down to the set of A and B that have equal sums.
• an SQL query that computes the sum of A and B by the country code can be run in advance, or as a sub-query, of the constraint query and only those countries that have equal sums need be fetched in the actual constraint query that returns individual events.
• constraint solving may be performed on the query results from each constructed query. Aspects of the constraint solving have been previously described herein, and for brevity sake, will not be described again.
• system 1500 includes processor 1501, memory 1503, and devices 1505-1508 via a bus or an interconnect 1510.
• Processor 1501 may represent a single processor or multiple processors with a single processor core or multiple processor cores included therein.
• Processor 1501 may represent one or more general-purpose processors such as a microprocessor, a central processing unit (CPU), or the like. More particularly, processor 1501 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets.
• CISC complex instruction set computing
• RISC reduced instruction set computing
• VLIW very long instruction word
• System 1500 may further include IO devices such as devices 1505-1508, including network interface device(s) 1505, optional input device(s) 1506, and other optional IO device(s) 1507.
• Network interface device 1505 may include a wireless transceiver and/or a network interface card (NIC).
• the wireless transceiver may be a WiFi transceiver, an infrared transceiver, a Bluetooth transceiver, a WiMax transceiver, a wireless cellular telephony transceiver, a satellite transceiver (e.g., a global positioning system (GPS) transceiver), or other radio frequency (RF) transceivers, or a combination thereof.
• the NIC may be an Ethernet card.
• a mass storage may also couple to processor 1501.
• this mass storage may be implemented via a solid state device (SSD).
• SSD solid state device
• the mass storage may primarily be implemented using a hard disk drive (HDD) with a smaller amount of SSD storage to act as a SSD cache to enable non-volatile storage of context state and other such information during power down events so that a fast power up can occur on re-initiation of system activities.
• HDD hard disk drive
• a flash device may be coupled to processor 1501, e.g., via a serial peripheral interface (SPI).
• SPI serial peripheral interface
• Computer-readable storage medium 1509 may also be used to store the some software functionalities described above persistently. While computer-readable storage medium 1509 is shown in an exemplary embodiment to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The terms “computer-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media, or any other non- transitory machine-readable medium.
• Processing module/unit/logic 1528, components and other features described herein can be implemented as discrete hardware components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices.
• processing module/unit/logic 1528 can be implemented as firmware or functional circuitry within hardware devices.
• processing module/unit/logic 1528 can be implemented in any combination hardware devices and software components.
• system 1500 is illustrated with various components of a data processing system, it is not intended to represent any particular architecture or manner of interconnecting the components; as such details are not germane to embodiments of the present disclosure. It will also be appreciated that network computers, handheld computers, mobile phones, servers, and/or other data processing systems which have fewer components or perhaps more components may also be used with embodiments of the disclosure.
• processing logic that comprises hardware (e.g. circuitry, dedicated logic, etc.), software (e.g., embodied on anon-transitory computer readable medium), or a combination of both.
• processing logic comprises hardware (e.g. circuitry, dedicated logic, etc.), software (e.g., embodied on anon-transitory computer readable medium), or a combination of both.
• Embodiments of the present disclosure are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of embodiments of the disclosure as described herein.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Software Systems (AREA)
• Multimedia (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=85412941

Family Applications (1)

Country Status (3)

Family Cites Families (4)

• 2022
  • 2022-09-02 WO PCT/US2022/042519 patent/WO2023034603A1/en not_active Ceased
  • 2022-09-02 JP JP2024514016A patent/JP2024536720A/ja active Pending
  • 2022-09-02 EP EP22865628.6A patent/EP4397014A4/de active Pending

Also Published As

Similar Documents

Legal Events