Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F8/00—Arrangements for software engineering
  • G06F8/30—Creation or generation of source code
  • G06F8/34—Graphical or visual programming
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/18—File system types
  • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
  • G06F16/1815—Journaling file systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F8/00—Arrangements for software engineering
  • G06F8/40—Transformation of program code
  • G06F8/41—Compilation
  • G06F8/43—Checking; Contextual analysis
  • G06F8/433—Dependency analysis; Data or control flow analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
  • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
  • G06F11/3466—Performance evaluation by tracing or monitoring
  • G06F11/3476—Data logging
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/11—File system administration, e.g. details of archiving or snapshots
  • G06F16/116—Details of conversion of file system types or formats
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/13—File access structures, e.g. distributed indices
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/24—Querying
  • G06F16/245—Query processing
  • G06F16/2455—Query execution
  • G06F16/24568—Data stream processing; Continuous queries
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/28—Databases characterised by their database models, e.g. relational or object models
  • G06F16/284—Relational databases
  • G06F16/285—Clustering or classification
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
  • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
  • G06F21/55—Detecting local intrusion or implementing counter-measures
  • G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
  • G06F2201/865—Monitoring of software

Definitions

• the present description relates to computer-implemented methods, data processing systems and machine-readable hardware storage devices for generating key-specific logs from execution of executable logic that is currently applied to one or more keyed data items in a stream of data items, e.g., data items coming in via a network connection from networked applications.
• the generating of the log record specifying the execution of those one or more portions of the executable logic for that given value of the key includes: specifying in the generated log record an outcome of the execution of those one or more portions of the executable logic, specifying variables that are read or set upon the execution of those one or more portions of the executable logic, or identifying those one or more portions of the executable logic that are executed.
• the method further includes: in response to completing the processing of the one or more of the data items that are each associated with the given value of the key, updating the state data to specify a current state for that value of the key.
• the method further includes: dynamically determining metrics indicative of which nodes of the chart are traversed and how many times the traversed nodes are traversed by the data items.
• the method further includes: comparing the selected first one of the data items of the stream currently being processed with a predetermined reference data item; determining if a deviation exists between the first one of the data items of the stream currently being processed and the predetermined reference data item or if such a deviation does not exist; and outputting an indication whether the first one of the data items of the stream currently being processed is in agreement with the predetermined reference data item based on the determining if a deviation exists between the first one of the data items of the stream currently being processed and the predetermined reference data item or if such a deviation does not exist.
• a data processing system for identifying one or more portions of executable logic that are executed in processing one or more data items that are each associated with a given value of a key, wherein a specification represents the executable logic, wherein the specification is associated with the key and wherein states of the specification are maintained for respective values of the key, the data processing system including one or more processing devices; and one or more machine-readable hardware storage devices storing instructions that are executable by the one or more processing devices to perform operations including: accessing the specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; intermittently receiving, over an input device or port, data items of a stream of data, with one or more of the data items each being associated with the given value of the key; and generating a log record for the given value of the key, with the log record specifying the one or more portions of the executable logic that are executed in processing the one or more of the data items that are
• a subset of log records associated with a particular key do not include the key field.
• the operations further include writing log records in a particular group associated with a particular key to a file at an end of predefined time interval.
• a log record is a structured data record with a first field for storing data representing a state of the specification for a particular value of the key and a second field for storing data specifying which portions of the executable logic represented by the specification have been executed, from a time at which execution of the executable logic is started to at a current time, for that particular value of the key.
• the operations further include writing log records in a particular group associated with a particular key to a file at an end of predefined time interval.
• the operations further include parsing the log records to identify which nodes of the chart are traversed and which nodes of the chart are not traversed.
• FIG. 6 is a diagram of an example process in key-based logging for processing of structured data items with executable logic.
• the charts can be stored in data storage 26.
• Storage devices providing the data source 12 may be local to the execution environment 14, for example, being stored on a storage medium connected to a computer running the execution environment 14 (e.g., hard drive 18), or may be remote to the execution environment 14, for example, being hosted on a remote system (e.g., mainframe 20) in communication with a computer running the execution environment 14 over a local or wide area data network.
• a remote system e.g., mainframe 20
• a dataflow graph can be highly technical in nature in some cases. While written to achieve specific business ends, the underlying structure and construction of the graph is determined based upon technical considerations. For example, graph components may be selected to maximize reusability, or to support parallel processing. On the other hand, how and where a graph is used may be largely a business decision. Some of the parameters associated with such a parameterized dataflow graph can be used to enable business users to customize dataflow graphs without requiring the user to understand the technical complexities behind its implementation. The parameterized dataflow graphs simplify customization and facilitate reuse.
• database 116 includes a relational database, rather than an ICFF, as the goal of implementation of database 116 is to be able to query for sequences of events. To do so, the sytem transforms the stream of audit records into one or more records (e.g., ICFF records). Each record includes one or more indexable/searchable key fields and a collection of data representing a remainder of the audit events.
• user interface 124 includes an investigation interface, in which users can ask questions, such as "what happened when subscriber 12345 got event 7 on June 8th?" To answer these questions, the system queries the ICFF (e.g., in database 116) - using a server side module (e.g., Java, a graph and so forth) between user interface 124 and the ICFF in database 116.
• a server side module e.g., Java, a graph and so forth
• Ruleset called timestamp R This audit record type specifies when a ruleset is subject id, about to be called by the event.
• Each monitoring point (which are associated with point subject id, * links) has a condition and an expression.
• the event instance id, engine generates an audit record only when the point name, condition is true and computes the expression and point value records the resulting value.
• a timeout happens for a subject for a chart.
• event engine 102 generates the following audit records: an "event seen" audit record of type alarm, then an "alarm dispatch” audit record and then a "block exit - alarm” audit record. From these audit records, the system determines values for the following key fields: the event type, timestamp, and instance id. In this example, the system determines a value of the instance id field from the block exit - alarm audit record, e.g., rather than inferring a value of that field. In this example, the system collects these audit records in a blob and the key fields (e.g., event type, timestamp, instance id, etc.) are set by the system from the data seen (e.g., the audit records).
• the key fields e.g., event type, timestamp, instance id, etc.
• Audit Events This is a list of event types (e.g., numbers corresponding to or
• the event engine performs the calculation (e.g., in the above described audit groups) every time the event engine detects an event or an alarm - but only at the beginning of processing of each event or alarm (e.g., rather than on every audit record).
• the event engine saves the conditional audit calculations in a package of global variables and queries inside each of the auditing functions that can be conditional.
• the system generates metrics database 120 by post-processing the audit records from the ICFF (in database 116), e.g., rather than reading the audit records from memory in the event engine 102.
• ICFFs are not purged.
• the metrics database 120 can be recreated at any time including using different aggregation periods.
• a table in metrics database 120 has a cluster name column such that one database can be used for multiple clusters.
• a cluster includes an event engine instance.
• an event engine instance includes a specific realization of one or more specifications (for which one or more charts are executed by event engine 102) for a unique value of a key. That is, each subject ID is handled by exactly one event engine instance. So, for each unique key (e.g., subject ID), there will be exactly one event engine instance that has all of its charts. Audit records for a same cluster will have the same value of cluster name.
• the metrics database 120 can be queried (e.g., by a client device or the system described herein) to retrieve data specifying reasons that messages were not sent for a particular key (e.g., for each subscriber).
• the event engine is configured to log links taken (and/or components executed within a chart) and ruleset cases accessed and implemented via the event engine 102 (e.g., by producing audit records specifying which components and/or links are executed).

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Software Systems (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Computer Security & Cryptography (AREA)
• Computer Hardware Design (AREA)
• Quality & Reliability (AREA)
• Computational Linguistics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Debugging And Monitoring (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (9)

Applications Claiming Priority (4)

Publications (1)

Family

ID=66433293

Family Applications (1)

Country Status (9)

Cited By (1)

Families Citing this family (9)

Citations (5)

Family Cites Families (25)

• 2017
  • 2017-12-22 US US15/853,000 patent/US11055074B2/en active Active
• 2018
  • 2018-11-13 DE DE112018005452.7T patent/DE112018005452T5/de not_active Withdrawn
  • 2018-11-13 CN CN202111522670.6A patent/CN114416684B/zh active Active
  • 2018-11-13 AU AU2018365176A patent/AU2018365176B2/en active Active
  • 2018-11-13 CN CN201880081094.6A patent/CN111837121B/zh active Active
  • 2018-11-13 EP EP18839952.1A patent/EP3710969B1/en active Active
  • 2018-11-13 CA CA3082454A patent/CA3082454A1/en active Pending
  • 2018-11-13 SG SG11202003758RA patent/SG11202003758RA/en unknown
  • 2018-11-13 WO PCT/US2018/060670 patent/WO2019094905A1/en not_active Ceased
  • 2018-11-13 JP JP2020544375A patent/JP7009643B2/ja active Active
• 2021
  • 2021-05-11 US US17/317,285 patent/US12190083B2/en active Active
• 2022
  • 2022-01-11 JP JP2022002291A patent/JP7254975B2/ja active Active

Patent Citations (5)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events