WO2014035338A1

WO2014035338A1 - Visual analytics enabled control system application graphical development environment

Info

Publication number: WO2014035338A1
Application number: PCT/SG2012/000409
Authority: WO
Inventors: Ying Lee LOW; Kwok Wai Wong; Karunakaran LIJESH
Original assignee: Neutrinos Engineering Pte Ltd
Priority date: 2012-08-31
Filing date: 2012-10-31
Publication date: 2014-03-06

Abstract

Embodiments in accordance with the present disclosure are directed to a graphical programming, development, or test architecture, platform, or environment for creating and deploying custom or customizable visual analytics applications. Multiple aspects of the present disclosure are directed to a visual analytics graphical programming, development, or test architecture, platform, or environment configured for automatically providing visual analytics capabilities to automation or control systems such as SCADA (Supervisory Control and Data Acquisition) systems.

Description

VISUAL ANALYTICS ENABLED CONTROL SYSTEM APPLICATION GRAPHICAL DEVELOPMENT ENVIRONMENT

Technical Field

The present disclosure relates generally to systems and methods for visual analytics or video content analysis. More particularly, aspects of the present disclosure are directed to a graphical programming, development, or test architecture, platform, or environment for creating and deploying custom or customizable visual analytics applications. Multiple aspects of the present disclosure are directed to a visual analytics graphical programming, development, or test architecture, platform, or environment configured for automatically providing visual analytics capabilities to automation or control systems such as SCADA (Supervisory Control and Data Acquisition) systems.

Background

Graphical programming environments for developing software applications for controlling various types of control system processes (e.g., industrial processes) have existed for a number of years. For instance, National Instruments Corporation provides commercially available graphical programming control system application development environments based upon their LabVIEW platform. Unfortunately, while conventional control system graphical programming environments support a wide variety of sense / measurement / actuation functions or operations, conventional control system graphical programming environments are undesirably limited in the types of functions they support in relation to video. Summary

In accordance with an aspect of the present disclosure, a graphical programming architecture configurable for developing visual analytics (VA) enabled control system applications, the architecture includes a control unit comprising a set of visual analytics modules (VAMs) couplable to at least one video source configured for capturing video information within an environment, each VAM within the set of VAMs comprising a visual analytics engine configured for analyzing captured video information received from the at least one video source, performing visual analytics operations upon captured video information to determine whether visual analytics events have occurred within the environment, and selectively generating at least one of an alert, a notification, and visual analytics metadata corresponding to a visual analytics event; and a host system comprising a computer system couplable to the control unit, the host system providing a graphical control system development environment that includes a set of graphical programming graphical user interfaces (GUIs), a set of control system application development tools, and a VA library, the VA library providing a set of graphical visual analytics objects corresponding to visual analytics functions that can be performed by the of VAMs. VA enabled control system application development occurs by way of control system application developer interaction with the host system's graphical programming GUIs and control system application developer selection of graphical visual analytics objects. The architecture can further include a visual analytics rule management interface couplable to the VAM, the visual analytics rule management interface configured for defining and editing visual analytics rules in response to graphical user input. Performing visual analytics operations comprises determining whether variations in one or more categorical types of visual elements, objects, or regions within sequences of video images have occurred based upon visual analytics rules defined by way of the visual analytics rule management interface.

A the visual analytics rule management interface can be configured for providing a visual analytics rule definition / modification GUI that provides: (a) a video scene window; (b) a visual event palette configured for receiving user input corresponding to user selection of one or more graphical visual analytics event identifiers corresponding to categorical types of visual analytics events, and facilitating user positioning of graphical representations of categorical types of visual analytics events within particular portions of the video scene window; and (c) a set of graphical elements configured for receiving user input directed to defining or modifying visual analytics rules corresponding to the graphical . representations of categorical types of visual analytics events positionable within the video scene window.

The set of graphical elements provided by the visual analytics rule definition / modification GUI can be configured for receiving user input corresponding to at least one of a rule name, a rule type, an event detection circumstance, textual output to be provided in response to occurrence of visual analytics events corresponding to a rule, a temporal schedule during which a rule is to remain actively processed by a VAM, and a visual object filtering function.

In accordance with a further aspect of the present disclosure, an architecture can further include at least one human-machine interface (HMI) configured for receiving visual analytics alerts, notifications, or metadata output by a VAM within the set of VAMs, and ; which is further configured for at least one of (a) receiving captured video information; (b) presenting visual control system state information corresponding to at least one control system corresponding to the environment; (c) issuing commands to control an operation performed by at least one control system performed by a control system corresponding to the environment; and (d) communicating control system state information corresponding to at least one control system corresponding to the environment to a destination external to the HMI following receipt of alerts, notifications, or metadata output by a VAM within the set of VAMs. The at least one HMI can include a computer system configured for executing a supervisory control application, the supervisory control application configured for controlling a process performed by at least one control system corresponding to the environment.

In accordance with another aspect of the present disclosure, the architecture's control unit further includees a set of sense / measurement / actuation (SMA) modules, each SMA module within the set of SMA modules couplable to an SMA device and configured for , performing at least one sensing, measurement, or actuation function within a control system corresponding to the environment; and the graphical control system development environment further includes an SMA library providing a set of graphical SMA objects . ■ corresponding to, SMA functions that can be performed by the set of SMA modules.

The graphical host system development environment can include or be, for instance, one 5 of a Lab VIEW based graphical programming environment and a Matlab based graphical programming environment. A VAM can include or be a module that can be removably coupled to a standard chassis, for instance, corresponding to a CompactRIO (cRIO) programmable automation controller.

10 In accordance with particular aspects of the present disclosure, an architecture further i includes at least one of a first set of VAMs and a first set of SMA modules disposable within a first environment corresponding to a first control system; and at least one of a : ¹ . second set of VAMs and a second set of SMA modules disposable within a second _, ·^■;■··. environment corresponding to a second control system, wherein the second control . 15 system is remote from the first control system. At least one of the first and second environments can be an indoor environment or an outdoor environment.

In accordance with an aspect of the present disclosure, a process for generating a visual analytics (VA) enabled control system application includes providing a control system

20 graphical programming environment including a sense / measurement / actuation (SMA) library and a set of graphical programming graphical user interfaces (GUIs); providing a visual analytics library corresponding to the control system graphical programming environment, the visual analytics library including a set of graphical objects corresponding to visual analytics functions; defining a VA enabled control system

25 application in response to developer interaction with the set of graphical programming GUIs; and automatically generating program instructions corresponding to the VA enabled control system application.

In accordance with a further aspect of the present disclosure, a process for deploying and 30 executing a visual analytics (VA) enabled control system application include providing VA enabled control system elements including (a) a set of sense / measurement / actuation (SMA) modules configured for communicating with a set of SMA devices, (b) a set of VA modules configured for communicating with a set of video sources, and (c) a control system application configured for processing information received from the set of SMA modules and the set of VA modules; defining rules corresponding to the occurrence of control related VA events or VA event sequences (e.g., VA events or event sequences related to process control and/or process safety); determining whether a control related VA event or VA event sequence has occurred; and generating an alert corresponding to an occurrence of the VA event or VA event sequence. Brief Description of the Drawings

FIG. 1 is a block diagram of a graphical programming architecture 10 configured or configurable for developing visual analytics enabled control system applications in accordance with an embodiment of the present disclosure. FIG. 2 is a block diagram of a representative SCADA system based upon an architecture in accordance with an embodiment of the present disclosure.

FIG. 3 is a block diagram of a representative control unit in accordance with an embodiment of the present disclosure.

FIG. 4A is a schematic illustration of a representative rule management interface which includes a rule definition / modification GUI in accordance with an embodiment of the present disclosure. FIG. 4B is a schematic illustration of a representative client VA alert interface in accordance with an embodiment of the present disclosure, which is configured for presenting, outputting, or displaying alert information or data in response to the generation or triggering of a set of alerts corresponding to one or more visual analytics rules. FIG. 4C is a schematic illustration of a representative additional alert GUI in accordance with an embodiment of the present disclosure.

FIGs. 5A - 5D are schematic illustrations of representative graphical objects or icons for the particular virtual instrument (VI) graphical objects corresponding visual analytics commands directed to a real-time processor in accordance with an embodiment of the present disclosure.

FIG. 6A is a flow diagram of a process for generating portions of one or more VA enabled control system applications in accordance with an embodiment of the present disclosure.

FIG. 6B is a flow diagram of a process for deploying and executing a VA enabled control system application in accordance with an embodiment of the present disclosure.

Detailed Description

In the present disclosure, depiction of a given element or consideration or use of a particular element number in a particular FIG. or a reference thereto in corresponding descriptive material can encompass the same, an equivalent, or an analogous element or element number identified in another FIG. or descriptive material associated therewith. The use of "/" in a FIG. or associated text is understood to mean "and/or" unless otherwise indicated. The recitation of a particular numerical value or value range herein is understood to include or be a recitation of an approximate numerical value or value range.

As used herein, the term "set" corresponds to or is defined as a non-empty finite organization of elements that mathematically exhibits a cardinality of at least 1 (i.e., a set as defined herein can correspond to a unit, singlet, or single element set, or a multiple element set), in accordance with known mathematical definitions (for instance, in a manner corresponding to that described in An Introduction to Mathematical Reasoning: Numbers, Sets, and Functions, "Chapter 11: Properties of Finite Sets" (e.g., as indicated ! on p.i 140), by Peter. J. ' Eccles, Cambridge University Press (1998)). In general, an ·. element of a¹ set ca include .or be a system, an apparatus, a device, a structure, an object, a process, a physical parameter, or a value depending upon the type of set under consideration.

Embodiments in accordance with the present disclosure are directed to a visual or graphical programming, development, modeling / simulation, and/or test architecture, platform, or environment configured or configurable for generating and deploying custom or customizable visual analytics or video content analysis applications. Various embodiments in accordance with the present disclosure are directed to a visual or graphical programming, development, modeling / simulation, and/or test architecture, platform, or environment configured or configurable for integrating or incorporating custom, customizable, or selectable visual analytics capabilities or functionality into automation, control, sensing, monitoring, measurement, data acquisition, and/or test system applications, for instance, for SCADA (Supervisory Control and Data Acquisition) system applications.

For purpose of brevity and ease of understanding, (a) visual or graphical programming, development, modeling / simulation, and/or test architectures, platforms, or environments are referred to in the description that follows as "graphical programming environments"; and (b) automation, control, sensing, sensing, monitoring, measurement, data acquisition, and/or test systems and corresponding system applications are respectively referred to herein as control systems and control system applications. In the context of various embodiments in accordance with the present disclosure, a control system application programmer or developer can be defined as an individual that interacts with a visual analytics enabled graphical control system application programming environment to program or develop a set of control system applications having one or more types of visual analytics functionality relevant to, defined for / by, or customized / customizable for one or more clients (e.g., based upon or tailored in accordance with client needs). A control system application programmer or developer can thus generate client-targeted or client-specific visual analytics enabled control system applications.

In multiple embodiments, one or more types of visual analytics capabilities or functions can be selectively or selectably integrated or incorporated into a set of control system applications in response to control system application developer or programmer input directed to a set of visual analytics graphical user interfaces (GUIs) and/or a set of graphical visual analytics objects. In such embodiments, visual analytics GUIs, graphical visual analytics objects, and/or a set of visual analytics functions or operations can be provided in association with, form portions of, or be integrated into a commercially available or conventional graphical programming environment that is suitable for developing control system applications, for instance, a LabVIEW based platform (National Instruments Corporation, Austin, TX USA). Graphical visual analytics objects and/or visual analytics functions can be provided in association with a visual analytics library and/or toolkit configured for providing visual analytics capabilities to or augmenting the visual analytics capabilities of a commercially available graphical programming environment.

Several embodiments in accordance with the present disclosure can thus (a) provide visual analytics capabilities to a commercially available graphical programming environment that lacks inherent, standard, or built-in visual analytics capabilities; or (b) modify, extend, or enhance visual analytics capabilities provided by a commercially available graphical programming environment having inherent, standard, or built-in visual analytics capabilities that are undesirably limited or which cannot be readily integrated into one or more types of control system applications. For instance, multiple embodiments in accordance with the present disclosure can provide a commercially available graphical control system application development environment, such as a LabVIEW application development environment associated with a National Instruments reconfigurable I/O (RIO) control platform (e.g., CompactRIO or cRIO), with graphically programmable visual analytics capabilities that can be integrated into control system applications such as SCADA applications. Aspects of control system application development based upon the cRIO platform are described in detail in the "NI Lab VIEW for CompactRIO Developer's Guide," published by National Instruments.

FIG. 1 is a block diagram of a graphical programming architecture 10 configured or configurable for developing visual analytics enabled control system applications (hereafter "VA-control applications") in accordance with an embodiment of the present disclosure. In an embodiment, the architecture 10 includes a host system 100; at least one control unit 200; at least one system of interest, system under consideration, system to be controlled, or system in which control operations are intended to occur 300; typically at least one visual analytics enabled human-machine interface (VA-HMI) 350; possibly one or more additional, standard / conventional, or adjunct HMIs 360; typically a client supervisory control interface 370; an adjunct / client visual analytics management interface (client VA management interface) 500; and a set of servers 700. Each element of the architecture 10 is configured for wireless and/or wire-based communication of information, signals, and/or data by way of a network infrastructure 20 that includes set of networks, such as one or more controller area networks (CANs), local area networks (LANs), wide area network (WANs), the Internet, and/or one or more other types of networks (e.g., a mobile telephone network). The host system 100 includes a computer system (e.g., a desktop or laptop computer) configured for providing a graphical control system development environment 110 that can execute on the host system 100, and which provides or includes a set of graphical programming GUIs 120; a set of control system application development tools 130; a sense / measurement / actuation (SMA) library 140; and a visual analytics (VA) library 150 for facilitating the development of a set of VA-control applications 160. Control system application programming, or VA-control application development, occurs by way of control system application developer interaction with the host system's graphical programming GUIs 120. The control unit 200 is configured for controlling one or more types of systems, subsystems, apparatuses, devices, or elements disposed relative to or within the system of interest 300. In multiple embodiments, the control unit 200 includes a control module 250; possibly one or more SMA modules (SMAMs) 210; and one or more visual analytics modules (VAMs) 400. Each SMAM 210 is coupled to a set of SMA devices 310 (e.g., one or more pressure sensors, temperature sensors, motion sensors, motion actuation devices, etc...) configured for providing or performing particular types of SMA functions or operations (e.g., correspondingly, pressure sensing, temperature sensing, motion sensing, or motion actuation, etc ..) within the system under consideration 300. Each VMAM 400 is coupled to at least one video source (e.g., a camera) 320 configured for capturing and providing or outputting video data within a video source field of view corresponding to the system of interest 300. The control module 250 is configured for managing or coordinating control related operations, functions, or processes performed by each SMAM 210 and each VAM 400, as further detailed below.

The VA-HMI 350 includes or is provided by a computer system configured for providing a client-side visual analytics enabled control system interface (client VA-control interface) 352, which can include a visual analytics control system information GUI that can output control system related visual information. In multiple embodiments, the client VA-control interface 352 is configured for selectively outputting (a) video signals, streams, or images captured by one or more video sources 320 positioned relative to or within the system of interest 300, and (b) visual analytics alert, notification, message, and/or metadata information corresponding thereto. The additional or standard HMI 360 includes or is provided by a computer system configured for providing an additional, standard / conventional, or adjunctive client-side control system interface (client control interface) 362 in which the presentation of visual analytics information is disabled, substantially excluded, or excluded. The additional or standard client control interface 362 can include a GUI configured for outputting data values or signal parameters / levels correlated with or corresponding to the operation of one or more SMA devices 310 disposed relative to or within the system of interest 300. In some embodiments, the client VA-control interface 352 is configured for providing visual analytics related control system information as well as one or more types of non- visual analytics control system information provided by the client control interface 362. As a result, in certain . embodiments, the client VA-control interface 352 effectively includes on subsumes aspects of the additional or standard client control interface 362, and thus in such embodiments the VA-HMI 350 effectively includes or subsumes aspects of the additional or standard HMI 350, which in a number of embodiments can be omitted or excluded.

The client supervisory control interface 370 includes or is provided by a computer system configured for providing a client supervisory control application 372 that can supervise, manage, or coordinate the selective or selectable (a) output of visual or graphical state and/or alert information, signals, or data corresponding to a system of interest 300 or a process performed thereby; (b) communication of state and/or alert information, signals, or data corresponding to a system of interest 300 or a process performed thereby to one or more destinations, networks, other systems, or devices; and/or (c) generation or issuance of supervisory client control commands to manage, affect, or control (e.g., adjust, interrupt, terminate, resume, or initiate) one or more operations, functions, or processes performed by or within portions of a system of interest 300. In some embodiments, the client supervisory control interface 370 can include or subsume portions of the VA-HMI 350 and/or the additional or standard HMI 360, and thus in such embodiments the client supervisory control interface 370 effectively includes or subsumes aspects of the VA- HMI 350 and/or the additional or standard HMI 350, one or each of which in a number of embodiments can be omitted or excluded.

The adjunct / client VA management interface 500 includes a computer system configured for selectively managing or monitoring operational aspects of, visual analytics functions provided by, and/or visual analytics alert generation by one or more VAMs 400. The VA management interface 500 can include a number of GUIs, as further detailed below. In some embodiments, the VA-HMI 350 and/or the client supervisory control system 370 can effectively include or subsume aspects of the adjunct / client visual analytics management interface 500, and thus in a number of embodiments, a separate adjunct / client visual analytics management interface 500 can be optional, omitted, or excluded.

The set of servers 700 is configured for providing computational and/or data storage resources to one or more portions of or systems / subsystems within the architecture 10. Depending upon embodiment details, servers 700 can be local, remote, or distributed.

A very wide variety of stand-alone, local, networked, or distributed monitoring, automation, or control systems can be based upon an architecture 10 in accordance with an embodiment of the present disclosure. Aspects of representative systems based upon the architecture 10 are described in detail hereafter.

FIG. 2 is a block diagram of a representative SCADA system 50 based upon an architecture 10 in accordance with an embodiment of the present disclosure. In an embodiment, the SCADA system 50 includes at least one instrumentation network 20a and/or at least one remote network 20b, and also a management, in-plant, or facility- internal network 20c. Such networks 20a-c can be configured for wire-based and/or wireless communication by way of an adjunct or support network infrastructure 20d, which can include a number of adjunct or support networks and/or network communication devices.

The instrumentation network 20a includes a control unit 200a coupled to one or more video sources 320 and one or more SMA devices 310, which are disposed relative to or within portions of an instrumentation system Of interest 300a; and a network communication apparatus or device (e.g., a wireless network communication transmitter / receiver) 22a configured for communicating control-related information, signals, or data (e.g., including visual analytics control information) between the control unit 200a and the management / in-plant network 20c. The instrumentation network 20a can further include one or more additional types of control system apparatuses, devices, or elements such as a programmable logic controller (PLC) 80. The instrumentation network 20a can also¹ include a set of network and/or communication resources 710a, such as an automated communication terminal unit.

The instrumentation system 300a can include or be essentially any type of instrumentation system, such as within the context of essentially any type of data acquisition, process control, automation (e.g., human procedure or manufacturing automation), or other system 300 that is located within or proximate to a plant / facility or plant / facility campus. As a non-limiting representative example, the instrumentation system 300a can include portions of one or more heat exchangers within a power plant or other type of industrial facility (e.g., a chemical processing plant). Such a system 300a can include a number of SMA devices 310 such as temperature sensors and pressure sensors configured for sensing or monitoring aspects of heat exchange process performance, control, and/or safety, in a manner understood by one of ordinary skill in the relevant art. The system 300a can further include a set of video cameras 320 configured for acquiring video signals or streams corresponding to particular heat exchanger structural elements or areas. Visual analytics operations can be performed upon captured video signals or streams to determine whether one or more types of heat exchange process control and/or safety events or circumstances have occurred or are occurring, which can involve the automatic or automated processing or analysis of video signals or streams by a VAM 400 to determine or indicate the occurrence or likely occurrence of particular types of visual analytics events or visual analytics event sequences. Such visual analytics events or visual analytics event sequences can indicate directly or by way of further processing (e.g., by a client computer system corresponding to a VA-HMI 350) the occurrence, existence, or likely occurrence / existence of a variety of heat exchange process control or safety related events, such as whether:

(a) personnel are approaching or proximate to particular heat exchanger areas or structures, where such personnel can be categorized as (i) expected, authorized, or likely authorized personnel such as maintenance / inspection personnel, whose presence is temporally expected in accordance with a maintenance schedule stored in a database, and possibly for whom a visual analytics based authorization likelihood can be facilitated by way of determining how long a given individual_;:; remains present or stationary at specific heat exchanger structures within a prescribed maintenance / inspection sequence, or whether the individual is wearing appropriate attire (e.g., a uniform having particular colors, and/or a hard hat), or (ii) unauthorized, likely unauthorized, or non-maintenance

/ inspection personnel (e.g., trespassers or vandals);

(b) one or more objects (e.g., tools or process control devices) have been introduced into, removed from, or altered within a spatial region, area, or environment external to the heat exchanger(s); and/or

(c) a video source 320 has become damaged, degraded, dysfunctional, or pointed toward an unexpected or unintended visual field.

In association with the occurrence of or in response to one or more types of visual analytics events- and/or heat exchange process control and/or safety events, alerts, notifications, or messages can be automatically communicated to the management / in- plant / in-facility network 20c, and/or a heat exchanger process can be adjusted, interrupted, terminated, or resumed.

The remote network 20b includes a control unit 200b coupled to at least one video source 320 and possibly one or more SMA devices 310, which are disposed relative to or within portions or a remote system of interest 300b; and a network communication apparatus or device 22b configured for communicating control-related information, signals, or data (e.g., including visual analytics control information) between the control unit 200b and the management / in-plant network 20c. The remote network 20b can include a server 700b, for instance, configured for storing or selectively storing video signals, streams, or data; and a set of network and/or communication resources 710b, such as an automated communication terminal unit (e.g., a remote terminal unit (RTU)).

The remote system of interest 300b can include or be essentially any type of system that is offsite, non-local, remote, or distant with respect to the geospatial location of the management / in-plant / in-facility network 20c, and within or upon which the performance of one or more types of control operations Or functions is desired or intended. Therefore^ the remote system of interest 300b can include or be essentially any type of remotely located data acquisition, process control, automation (e.g., human procedure or manufacturing automation), or other type of system 300.

As a non-limiting representative example, the remote system 300b can include one or more portions of a pipeline configured for transporting a chemical substance such as a fossil fuel product (e.g., crude oil) or a petroleum or petrochemical product. A number of SMA devices 310 such as pressure, temperature, and/or vibration sensors can be coupled to portions of the pipeline. Additionally, one or more video cameras 320 can be positioned to capture video signals or streams of portions of the pipeline, such that visual analytics operations can be performed upon captured video signals or streams to determine whether one or more types of pipeline control and/or safety events or circumstances have occurred. The determination of the occurrence of pipeline control and/or safety events can involve the automatic or automated processing or analysis of video signals or streams by a VAM 400 to determine or indicate the occurrence or likely occurrence of particular types of visual analytics events or visual analytics event sequences, which can indicate directly or by way of further processing the occurrence, existence, or likely Occurrence / existence of a wide variety of pipeline control or safety related events, such as whether:

(a) personnel and/or objects (e.g., vehicles) are approaching or proximate to particular pipeline areas or structures, where such personnel or objects can be categorized as authorized 7 expected authorized / likely authorized or unauthorized / likely unauthorized / non-maintenance or inspection personnel or objects;

(b) a remote video source 320 has become damaged, degraded, dysfunctional, or pointed toward an unexpected or unintended visual field;

(c) a pipeline containment leak, rupture, or spill is visually indicated or evident (e.g., by way of determining whether spatial regions within video scenes corresponding to pipeline structures or ground areas proximate to the pipeline have become visually distorted or obscured over time in a manner that would be expected or likely for a type of substance that the pipeline carries); and/or (d) pipeline-external leak, rupture, or spill conditions are improving, stabilizing, or worsening (e.g., by way of automatically determining and tracking video scene changes over time, which can indicate the size, growth, or diminishment of (i) a substance escaping from the pipeline, or (ii) a consequence of the pipeline leak, rupture, or spill (e.g., flames).

In association with the occurrence of or in response to one or more types of visual analytics events and/or pipeline control and/or safety events, one or more alerts, notifications, or messages can be automatically communicated to the management / in- plant / in-facility network 20c, and/or a pipeline process can be adjusted, interrupted, terminated, or resumed. The management / in-plant / in-facility network 20c includes at least one VA-HMI 350 and/or at least one standard HMI 360, an adjunct / client VA management interface 500, one or more servers 700c, and possibly a set of network and/or communication resources (e.g., network data storage resources) 710c. In association with or in response to the receipt of one or more types of visual analytics event alerts, corresponding metadata, and possibly captured video signals, streams, or sequences, a client supervisory control interface 370 associated with or corresponding to the VA-HMI 350 and possibly additionally associated with or corresponding to the HMI 360 (e.g., by way of a client supervisory control application 372 in communication with the VA-HMI 350 and/or the standard HMI 360, or developed in association with or as part of the client VA-control interface 352 and/or the standard client control interface 362) can process or analyze such visual analytics event alerts, metadata, and/or captured video signals, streams, or sequences and selectively (a) manage or direct the presentation of visual or graphical control system or process state and/or alert information, signals, or data by way of one or more display devices; (b) communicate control system or process state and/or alert information, signals, or data to one or more destinations, networks, systems, or devices (e.g., by way of short message service (sms) messages directed to particular mobile telephone numbers, or automated voice notifications directed to a set of mobile and/or landline telephone ' numbers, or a set of e-mails); and/or (c) issue a set of supervisory client control commands to manage, affect, or control (e.g., adjust, interrupt, terminate, resume, or initiate) one or more operations, functions, or processes performed by or within portions of a system of interest 300.

FIG. 3 is a block diagram of a representative control unit 200 in accordance with an embodiment of the present disclosure. In an embodiment, the control unit 200 includes a control module 250 in which a processing unit or target processor 252, a set of communication resources 260, and a memory 270 resides. The processing unit 252 can include a microprocessor or microcontroller configured for executing program instructions stored in the memory 270, where such program instructions include a control module Or target executable visual analytics control application (VA control executable) 290. The communication resources 260 can include hardware, firmware, and/or software resources for communicating with systems, apparatuses, devices, elements, or interfaces internal and/or external to the control module 250. In various embodiments, the communication resources include a set of SMAM communication resources 262 configured for communicating with SMAMs 210; a set of VAM communication resources 264 configured for communicating with VAMs 400; and a set of network communication resources 266 configured for information, signal, or data transfer with one or more types of networks (e.g., in accordance with a standard network protocol such as TCP/IP). In some embodiments, the control module's network communication resources 266 can be configured for facilitating communication between the control module 250 and one or more client VA-control interfaces 250 and/or client control interfaces 260.

The memory 270 additionally includes a number of program instruction resources (e.g., drivers) configured for facilitating or effectuating SMAM control operations and VAM control operations. For instance, the memory 270 can include a set of SMAM drivers 272 configured for facilitating or effectuating SMA control related operations or functions in a manner understood by one of ordinary skill in the relevant art. Furthermore, the memory 270 can include a set of visual analytics configuration drivers 280, a set of visual analytics alert drivers 282, a set of video and/or image acquisition drivers 284, and a set of video source information drivers 286 configured for facilitating visual analytics based control related operations or functions. In addition to the aforementioned VA-control executable 290, the memory 270 can also include an operating system 292, in a manner understood by one of ordinary skill in the relevant art.

As indicated in FIG. 3, a VAM 400 includes a video analytics engine or processing module 410 configured for analyzing one or more video streams, performing or applying particular types of visual analytics operations or functions upon such video streams, and selectively generating visual analytics alerts corresponding thereto. In general, the video analytics engine 410 is configured for analyzing sequences of video images, determining whether particular types of visual analytics events corresponding to the presence or absence of or variations in one or more categorical types of visual elements, objects, or regions within the video images, and generating alerts / notifications / messages and associated metadata corresponding to such events, in a manner understood by one of ordinary skill in the relevant art. Metadata can include information such as a time / date stamp, a video or image source identifier (ID), one or more object IDs, object coordinates, object property information, and object behavior information. Metadata can include, for instance, textual data (e.g., XML-based data). Metadata can be communicated external to the VAM 400, such as to a memory or a database, for instance, corresponding to one or more of the host system 100, the VA-HMI 350, the client supervisory control interface 370 (e.g., such that the metadata can be processed or analyzed by a client supervisory control application 372), a server 700, or another destination.

In several embodiments, the VAM 400 also includes a set of VAM network communication resources 420 separate from the control module's network communication resources 234, which can facilitate network communication with one or more systems, apparatuses, devices, or interfaces separate from those configured for information, signal, or data transfer by way of the control module's network communication resources 234. In a number of embodiments, the VAM network communication resources 420 are configured for information, signal, or data transfer with an adjunct / client VA management interface 500, for instance, in accordance with a standard network protocol (e.g., the TCP/IP protocol). The adjunct / client VA management interface can thus establish, manage, or monitor visual analytics information generated by one or more VAMs 400, as further described below.

In a representative embodiment, the control unit 200 can be implemented by way of a National Instruments cRIO system or platform in which the control module 250 includes a cRIO real-time controller that provides (a) a real time processor 252 coupled to an FPGA which is configured to provide control module communication resources 260; and (b) a real-time operating system (RTOS) configured for managing the execution of program instructions associated with VA-control executables to facilitate or enable realtime control operations or functions corresponding to one or more systems of interest 300. Any given SMAM 210 can include essentially any type of standard cRIO I O module, which can be matingly coupled or removably inserted into a cRIO chassis or frame assembly or structure. A VAM 400 can include a Neutrinos Intelligent Video Analytics Module (IV AM) (Neutrinos Engineering Pte Ltd., 5 Harper Road #04-02, Singapore) that can be matingly coupled or removably inserted into the cRIO chassis assembly (e.g., adjacent to a standard cRIO I/O module), and which includes a visual analytics / video analytics engine based upon a digital media processor (DMP) such as a Texas Instruments TMS320DM6437 Digital Media Processor (Texas Instruments, Dallas, TX USA). As indicated above, the architecture 10 can include or provide an adjunct or client VA management interface 500 that facilitates or enables client management or selection of particular categorical types of visual analytics processing operations or functions performed by one or more VAMs, and client identification or selection of corresponding visual analytics alert generation criteria or conditions. As indicated in FIG. 3, in some embodiments the VA management interface 500 provides a client visual analytics configuration management interface (client VA configuration interface) 510 and a client

1 visual analytics alert presentation (client VA alert interface) 560. Each of the client VA configuration! interface 510 and the client VA alert interface 560 can include a set of corresponding GUIs. < For instance, the client VA configuration interface 510 can include a VAM parameter configuration GUI configured for receiving user (e.g., control system application developer or client) input that specifies particular VAM parameters such as one or more VAM network addresses (e.g., IP addresses). The client VA configuration interface 510 can further include one or more visual analytics rule management interfaces, as detailed hereafter. In general, a visual analytics rule can be defined as a set of visual analytics events (e.g., a single visual analytics event, a temporally segregated sequence of visual analytics events, or temporally overlapping or substantially concurrent visual analytics events) that must occur or be detected or satisfied in order for the VAM 400 to generate a visual analytics alert. Any given visual analytics event can be defined as the occurrence of a particular categorical type of visual incident, situation, or phenomenon, the presence of which in relation to multiple video scenes, images, or frames indicates the presence or absence of or variations in one or more categorical types of visual elements, objects, or regions within the video images. Various types of visual analytics events can be supported, provided, or defined, for instance, "tripwire" events; "subject or object enters" events; "subject or object inside" events; "subject or object outside" events; "subject or object exits" events; "subject or object appears" events; "subject or object disappears" events; "object left behind" events; "object taken away" events; "loitering" events; "object size" based events, such as "object size change" events; "object trajectory / direction" events; and others. In association with determining whether a given type of visual analytics event has occurred, the VAM 400 can automatically categorize or attempt to categorize various types of visual elements, objects, or regions within one or more portions of video images, for instance, in accordance with categorizations such as "person," "group," "crowd," "animal," "vehicle," "unknown object," "equipment," "smoke," "fire," and "opaque substance." Furthermore, the VAM 400 can provide a number of subject or object s filtering ;f unctions _^such as i subject / object size filtering, shape' filtering, speed filtering >(ejg:,i "subje'ct walking!'', or "subject running"), and object trajectory / direction filtering; and/or image processing functions, such as low-light or nighttime image enhancement; and temporal measurement functions, such as a time period or duration that one or subjects or objects met the conditions of one or more visual analytics events.

FIG. 4A is a schematic illustration of a representative rule management interface provided by an embodiment in accordance with the present disclosure. In an embodiment, the rule management interface includes a rule generation / modification GUI 520, which includes a video scene window 522, a set of video playback and visual analytics rule - video image overlay controls 524, and a visual event palette 526 configured for receiving user input (e.g., drag and drop input) corresponding to user selection (e.g., graphical selection, such as by way of pointing device motion) of one or more graphical visual analytics event identifiers (e.g., icons or the like) representing categorical types of visual analytics events (e.g., "tripwire" events), and further configured for enabling or managing user positioning of graphical representations of such categorical types of events within particular portions or regions of a video image or video source field of view presented within the video scene window 522 (e.g., user positioning of a "tripwire" event across or along one or more portions of an environmental or physical structure within the video source's field of view, such as a road, driveway, or parking lot).

The rule generation modification GUI 520 further includes a number of graphical ; elements, objects, or widgets configured for receiving graphical or non-graphical user input directed to defining, selecting, or modifying visual analytics rules corresponding to the graphical representation(s) of the categorical type(s) of visual analytics event(s) positioned within the video scene window 522. Such graphical elements or objects can include a first text box 530 for receiving user input defining a rule name; a set of categorical rule type and event detection circumstance selection check boxes 532 for receiving user identification(s) of what categorical type(s) of objects within video images should be detected, and under what circumstance(s) or condition(s) such detection should occur; a second text box 534 for receiving user input defining: text' that is to be provided - or output in association with the generation of an alert corresponding to the rule under consideration; a schedule selection drop down menu 536 for receiving user input defining a temporal schedule (e.g., all the time, during business hours, or outside of business hours) during which a selected rule is to remain actively, repeatedly, or recurrently processed by the VAM 400.

The rule generation I modification GUI 520 can also include a filter dropdown menu 528 configured to receive user input defining or selecting a categorical type of object filtering function (e.g., object size, object shape, object trajectory / direction, or object speed) to be applied one or more types of objects to be detected in association with VAM processing of the visual analytics rule presented within the rule generation / modification GUI 520. In various embodiments, rules are stored in a memory within, local to, or accessible to a VAM 400 (e.g., rules can reside within an embedded memory and/or an on-board memory within the VAM 400, in a manner understood by one of ordinary skill in the relevant art). In some embodiments, rules can additionally be stored in a memory or database corresponding to the VAM's control module 250, the host system 100, the VA- HMI 350, the client supervisory control interface 370, and/or the adjunct / client VA management interface 500, one or more of which can execute or access a rule management interface (e.g., a rule generation / modification GUI 520 such as that described above) to selectively create, edit, and delete rules, and communicate rules to a VAM 400 in response to user input.

FIG. 4B is a schematic illustration of a representative client VA alert interface 560 in accordance with an embodiment of the present disclosure, which is configured for presenting, outputting, or displaying alert information or data in response to the generation or triggering of a set of alerts corresponding to one or more visual analytics rules. In an embodiment, the client VA alert interface 560 includes a set of video scene, image, or snapshot windows 562a-b configured for presenting video images for which alerts have been most recently been generated; an alert data table 564 configured for presenting alert data such as an alert ID, a time / date stamp, an alert description, and a rule type, and possibly other alert information; a set of alert text boxes 566 configured to present alert information corresponding to a current or most-recenfly generated alert; and a set of additional text boxes 568 configured for presenting video source information (e.g., an IP address and port number) and configured for receiving user input specifying a client user name and password.

In some embodiments, the VA alert interface 560 and/or a host system resident VA- control application 160 can have another alert GUI associated therewith configured for presenting alert information or data corresponding to the control module's network communication resources reception, retrieval, or communication / confirmation of alert generation. FIG. 4C is a schematic illustration of a representative additional, adjunctive, or auxiliary alert GUI (e.g., a host-side visual analytics alert GUI) 570 in accordance with an embodiment of the present disclosure. In an embodiment, the additional alert GUI 750 includes a graphical window (e.g., a scrollable child window) 572 in which particular alert information or data corresponding to current and recent alerts (e.g., an alert history) can be presented; and a set of text boxes 574 in which certain alert information corresponding to a current or most-recently generated alert can be presented.

As indicated herein, an architecture 10 in accordance with an embodiment of the present disclosure includes or provides a set of graphical visual analytics objects which a control system application developer can access and graphically manipulate (e.g., drag and drop), and/or to which the control system application developer direct input or commands (e.g., graphical selections and/or textual input) for providing particular types of visual analytics functionality to a set of visual analytics control applications, which can include a VA- control executable 290 and possibly one or more of a client VA control interface 362, a client VA configuration interface 510, a client VA alert interface 560, and a client ^:; supervisory control application 372. Graphical visual analytics objects can reside within; a visual analytics library 150.

Graphical visual analytics objects include graphical objects corresponding to particular control module visual analytics drivers, such as the visual analytics related drivers 280- 286 indicated above with reference to FIG. 3. Additionally, such graphical visual analytics objects can include graphical objects (e.g., drivers, driver-level, or driver type objects such as FPGA configuration code objects) corresponding to VAM communication resources 264, and/or visual analytics module network communication resources (e.g., VAM TCP/IP communication resources) 420.

In several representative embodiments involving or based upon a National Instruments cRIO real time controller, the set of graphical visual analytics objects includes graphical representations of National Instruments Virtual Instruments (Vis), such as the following representative types of VI graphical objects:

(a) a set of VI graphical objects corresponding to visual analytics commands directed to the real-time processor 252, including a "Configure VAM" VI graphical object corresponding to the VA configuration driver 280, the "Get VA Alert" VI graphical object corresponding to a VA alert driver 282, the "Get Snapshot" VI graphical object corresponding to an image acquisition driver, and a "Get VAM Info" VI graphical object corresponding to the video source information driver 286;

(b) one or more VI graphical objects corresponding to the communication resources FPGA 260, to facilitate or effectuate information, signal, or data transfer between the FPGA 260 and the VAM 400; and

(c) a set of VI graphical objects corresponding to particular VAM functions that can be directed to the VAM 400 by way of the VAM network communication resources 420, including a "Set VA Rule" VI graphical object and a "Get VA Alert" VI graphical object. FIGs. 5A - 5D are schematic illustrations of representative graphical objects or icons for the aforementioned VI , graphical objects corresponding to visual analytics commands directed to the real-time processor 252 in accordance with an embodiment of the present disclosure. More particularly, FIG. 5A is a schematic illustration of a representative "Configure VAM" VI graphical object in accordance with an embodiment of the present disclosure. In an embodiment, the "Configure VAM" VI graphical object facilitates the establishment or definition of initial VAM configuration parameters (e.g., user name, password, IP address, and/or other parameters). FIG. 5B is a schematic illustration of a representative "Get VA Alert" VI graphical object in accordance with an embodiment of the present disclosure. In an embodiment, the "Get VA Alert" VI graphical object facilitates the reception or retrieval of a visual analytics alert from a VAM 400 when an alert has occurred. An "Alert Info" array can include alert information such as one or more rule names, an alert type, and an alert description. FIG. 5C is a schematic illustration of a representative "Get Snapshot" VI graphical object in accordance with an embodiment of the present disclosure. In an embodiment, the "Get Snapshot" VI graphical object can facilitate the reception, retrieval, or reading of an image snapshot or image data frame from the VAM 400. Finally, FIG. 5D is a schematic illustration of a representative "Get VAM Info" VI graphical object in accordance with an embodiment of the present disclosure. In an embodiment, the "Get VAM Info" VI graphical object can facilitate the reception or retrieval of VAM device or device type information.

FIG. 6A is a flow diagram of a process 600 for generating portions of one or more VA enabled control system applications in accordance with an embodiment of the present disclosure. In an embodiment, a first process portion 602 includes providing a control system graphical programming environment, which includes a set of control system development tools and a set of graphical programming GUIs. A second process portion 604 includes providing an SMA library, which supports, corresponds to, or includes graphical SMA objects corresponding to SMA functions; and a third process portion 606 includes providing a VA library, which supports, corresponds to, or includes graphical visual analytics objects corresponding to visual analytics functions. A fourth process portion 608 includes defining a VA enabled control system application in response to developer interaction with the graphical programming GUIs and developer selection of graphical objects corresponding to the SMA library and the VA library. A fifth process portion 610 includes automatically generating one or more sets of program instructions (e.g., software) corresponding to a VA enabled control system application.

FIG. 6B is a flow diagram of a process 650 for deploying and executing a VA enabled control system application in accordance with an embodiment of the present disclosure. In an embodiment, a first process portion 652 includes providing or deploying VA enabled control system elements including (a) one or more SMA modules configured for communicating with at least one set of SMA devices; (b) one or more VA modules configured for communicating with at least one set of video sources; and (c) a VA enabled control system application configured for processing information, signals, or data received from the SMA module(s) and the VA module(s). A second process portion 654 includes identifying or defining (e.g., graphically defining, by way of a corresponding GUI) one or more rules corresponding to control related VA events or event sequences, which correspond to process control situations and/or control system safety. A third process portion 656 includes determining whether a process control and/or process safety related VA event has occurred; and a fourth process portion 658 includes generating and communicating a set of alerts, notifications, or messages to one or more devices or systems in response to an occurrence of a process control or process safety related VA event.

During or in association with the process 650 (e.g., in parallel or in sequence with the third and/or fourth process portions 656, 658), a control system application (e.g., the VA enabled control system application) can process information, signals, or data received from the SMA module(s), and perform SMA related control operations in a manner understood by one of ordinary skill in the relevant art. ti Aspects¹ of particular- embodiments of the present disclosure address at least one aspect^, problem', !-limitation, ! and/or disadvantage associated with exiting visual analytics architectures, systems, or processes and visual analytics application development therefor. While features, aspects, and/or advantages associated with certain embodiments 5 have been described in the disclosure, other embodiments may also exhibit such features, aspects, and/or advantages, and not all embodiments need necessarily exhibit such features, aspects, and/or advantages to fall within the scope of the disclosure. It will be appreciated by a person of ordinary skill in the art that several of the above-disclosed systems, components, processes, or alternatives thereof, may be desirably combined into0 . other different systems, components, processes, and/or applications. In addition, various modifications, alterations, and/or improvements may be made to various embodiments that are disclosed by a person of ordinary skill in the art within the scope and spirit of the present disclosure. 5 . For instance, while representative embodiments in accordance with the present disclosure have been described¹ * with reference to a Lab VIEW based application development environment, one of ordinary skill in the relevant art will understand that embodiments in accordance with the present disclosure are not limited to a Lab VIEW based application - development environment, but rather are applicable to essentially any type of graphical0 programming environment (for instance, a Math Works (e.g., MATLAB / Simulink) based graphical programming environment (Math Works, Inc., Natick, MA USA)) that is suitable for the development of sense, measurement, automation, control, sensing, monitoring, data acquisition, test, and/or related types of applications. 5 Moreover, in addition or as an alternative to the representative embodiments described above, embodiments in accordance with the present disclosure are applicable or extendable to graphical programming or development of non-control system applications or environments, such as intrusion detection or business, home, or public security applications; or public health or medical assessment applications (for instance, visual0 analytics based determination of whether one or more individuals within a hospital : : emergency room exhibit sigris of fever (e.g., as determined by way of a set of infrared cameras configured^, .to¹ identify elevated skin temperature) or certain types of trauma, and, ^; i selective generation of alerts to health practitioners or medical staff corresponding to such determinations). Furthermore in addition to the foregoing, one or more video sources 320 can be configured to provide or transfer pre-stored or previously captured video information. Such video sources 320 can be, for instance, DVD players, or a web browser that presents a pre-recorded video. A visual analytics application that receives such pre-stored video information can be configured to perform one or more types of "off-line" or non-real time , visual analytics operations or functions upon such pre-stored video information. '

The foregoing and other visual analytics architectures, systems, processes, and applications based upon or derived from the representative embodiments described herein are encompassed by the present disclosure, which is limited only by the following claims.

Claims

1. A graphical programming architecture configurable for developing visual analytics (VA) enabled control system applications, the architecture comprising:

a control unit comprising a set of visual analytics modules (VAMs) couplable to at least one video source configured for capturing video information within an environment, each VAM within the set of VAMs comprising a visual analytics engine configured for analyzing captured video information received from the at least one video source, performing visual analytics operations upon captured video information to determine whether visual analytics events have occurred within the environment, and selectively generating at least one of an alert, a notification, and visual analytics metadata corresponding to a visual analytics event; and

a host system comprising a computer system couplable to the control unit, the host system providing a graphical control system development environment that includes a set of graphical programming graphical user interfaces (GUIs), a set of control system application development tools, and a VA library, the VA library providing a set of graphical visual analytics objects corresponding to visual analytics functions that can be performed by the of VAMs.

2. The architecture of claim 1, wherein VA enabled control system application development occurs by way of control system application developer interaction with the host system's graphical programming GUIs and control system application developer selection of graphical visual analytics objects.

3. The architecture of claim 1, further comprising a visual analytics rule management interface couplable to the VAM, the visual analytics rule management interface configured for defining and editing visual analytics rules in response to graphical user input.

4. The architecture of claim 3, wherein performing visual analytics operations comprises determining whether variations in one or more categorical types of visual elements, objects; or regions; within sequences of video images have occurred based upon visual analytics rules defined by way of the visual analytics rule management interface.

5. The architecture of claim 3, wherein the visual analytics rule management interface is configured for providing a visual analytics rule definition / modification GUI that provides:

(a) a video scene window;

(b) a visual event palette configured for receiving user input corresponding to user selection of one or more graphical visual analytics event identifiers corresponding to categorical types of visual analytics events, and facilitating user positioning of graphical representations of categorical types of visual analytics events within particular portions of the video scene window; and

(c) a set of graphical elements configured for receiving user input directed to defining or modifying visual analytics rules corresponding to the graphical representations of categorical types of visual analytics events positionable within the video scene window.

6. The architecture of claim 5, wherein the set of graphical elements is configured for receiving user input corresponding to at least one of a rule name, a rule type, an event detection circumstance, textual output to be provided in response to occurrence of visual analytics events corresponding to a rule, a temporal schedule during which a rule is to remain actively processed by a VAM, and a visual object filtering function .

7. The architecture of claim 1, further comprising at least one human-machine interface (HMI) configured for receiving visual analytics alerts, notifications, or metadata output by a VAM within the set of VAMs, and which is further configured for at least one of

(a) receiving captured video information;

(b) presenting visual control system state information corresponding to at least one control system corresponding to the environment;

(c) issuing commands to control an operation performed by at least one control

^■ :■ ^■ system performed by a control system corresponding to the environment; and ^■ ( ί communicating control system state information corresponding to at least one , ; control systems corresponding to the environment to a destination external to the HMI following receipt of alerts, notifications, or metadata output by a VAM within the set of VAMs.

8. The architecture of claim 7, wherein the at least one HMI comprises a computer system configured for executing a supervisory control application, the supervisory control application configured for controlling a process performed by at least one control system corresponding to the environment.

9. The system of claim 7, wherein:

the control unit further comprises a set of sense / measurement / actuation (SMA) modules, each SMA module within the set of SMA modules couplable to an SMA device and configured for performing at least one sensing, measurement, or actuation function within a control system corresponding to the environment; and the graphical control system development environment further includes an SMA library providing a set of graphical SMA objects corresponding to SMA functions that can be performed by the set of SMA modules.

10. The architecture of claim 7, wherein the graphical host system development environment comprises one of a Lab VIEW based graphical programming environment and a Matlab based graphical programming environment.

1 1. The architecture of claim 10, wherein the VAM comprises a module that can be removably coupled to a chassis corresponding to a CompactRIO (cRIO) programmable automation controller.

12. The architecture of claim 9, wherein the control system comprises an industrial control system.

13. The architecture of claim 11, wherein the ; control system comprises a Supervisory Control and Data Acquisition (SCAD A) system.

14. The architecture of claim 11, further comprising:

at least one of a first set of VAMs and a first set of SMA modules disposable within a first environment corresponding to a first control system; and

at least one of a second set of VAMs and a second set of SMA modules disposable within a second environment corresponding to a second control system, wherein the second control system is remote from the first control system.

15. The architecture of claim 14, wherein at least one of the first and second environments is an indoor environment.

16. The architecture of claim 14, wherein at least one of the first and second environments is an outdoor environment.

17. A method for generating a visual analytics (VA) enabled control system application comprising:

providing a control system graphical programming environment including a sense / measurement / actuation (SMA) library and a set of graphical programming graphical user interfaces (GUIs);

providing a visual analytics library corresponding to the control system graphical programming environment, the visual analytics library including a set of graphical objects corresponding to visual analytics functions;

defining a VA enabled control system application in response to developer interaction with the set of graphical programming GUIs; and

automatically generating program instructions corresponding to the VA enabled

control system application.

18. A method for deploying and executing a visual analytics (VA) enabled control system application, comprising: providing VA. enabled control system elements including (a) a set of sense ^'/ measurement / actuation (SMA) modules configured for communicating with a set of SMA devices, (b) a set of VA modules configured for communicating with a set of video sources, and (c) a control system application configured for processing information received from the set of SMA modules and the set of VA modules;

defining rules corresponding to the occurrence of control related VA events or VA event sequences;

determining whether a control related VA event or VA event sequence has occurred; and

generating an alert corresponding to an occurrence of the VA event or VA event sequence.