CN114424164A - Updating detection models and maintaining data privacy - Google Patents

Updating detection models and maintaining data privacy Download PDF

Info

Publication number
CN114424164A
CN114424164A CN202080065674.3A CN202080065674A CN114424164A CN 114424164 A CN114424164 A CN 114424164A CN 202080065674 A CN202080065674 A CN 202080065674A CN 114424164 A CN114424164 A CN 114424164A
Authority
CN
China
Prior art keywords
model
module
model update
update
local node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202080065674.3A
Other languages
Chinese (zh)
Inventor
W·R·小帕特恩
E·I·克尔顿
马忆惠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/577,770 external-priority patent/US11216268B2/en
Priority claimed from US16/577,774 external-priority patent/US11188320B2/en
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of CN114424164A publication Critical patent/CN114424164A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/35—Creation or generation of source code model driven
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/60—Software deployment
    • G06F8/65—Updates

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A system for updating a detection model includes at least one local node including a monitoring module, a diagnostic module, and an evaluation module. The system receives at least one model update and analyzes the model update and the current model and data present in the local node and determines whether the update should be applied. In some embodiments, local nodes may generate model updates for use in other local nodes without sharing private data present in the local nodes.

Description

Updating detection models and maintaining data privacy
Technical Field
The present application relates generally to systems for evaluating and sharing detection models while maintaining data privacy and methods of use thereof.
Background
Analytical models for event detection are important to a range of fields and industries. For example, different analytical models are used to detect bank fraud, aid in regulatory compliance, and many other complex data-driven issues. Many areas require up-to-date models for accurate and timely event detection. In some areas (e.g., many types of fraud), third party agents are actively working to evade detection by current analytical models. Thus, there is a need for a system for updating a detection model that allows model updates to be distributed, analyzed, and implemented across multiple local nodes of the system in a rapid manner. Further, because manual creation of updated detection models can be a slow and time-consuming process, there is a need for a system that creates its own model updates from successful event detections and then distributes the created model updates to the rest of the system while maintaining data privacy of the location where the model updates were created.
Disclosure of Invention
One aspect of the invention provides a system for updating a detection model, comprising: at least one local node comprising: a monitoring module; a diagnostic module; an evaluation module; one or more current detection models; and a memory comprising instructions for execution by the at least one processor, the instructions configured to: receiving, by the monitoring module, a model update; determining, by the diagnostic module, the current monitoring model; determining, by the evaluation module, whether the model update should be applied to the current monitoring model; determining, by the evaluation module, whether the local node has permission to apply the model update; and updating, by the evaluation module, the current detection model with the model update.
In some embodiments, the system further comprises at least a second local node; a central module, wherein the monitoring module of each local node is in electronic communication with the central module; and a memory comprising instructions for execution by the at least one processor, the instructions configured to: creating a model update, the model update comprising: detecting, by the diagnostic module, a significant change in system data; determining from said diagnosis a list of all current test models involved in the testing step; analyzing, by the diagnostic module, system data involved in the detecting step; generating, by the diagnostic module, a model update; transmitting, by the monitoring module, the model update; distributing model updates, comprising: receiving, by the central module, the model update; analyzing, by the central module, a database of available models; determining, by the central module, a priority for the model update; determining, by the central module, which local nodes should receive the model update; and transmitting, by the central module, the model update; and updating at least one local node, including: receiving, by at least one monitoring module, a model update; determining, by at least one diagnostic module, the current monitoring model; determining, by at least one evaluation module, whether the model update should be applied to the current monitoring model; determining, by at least one evaluation module, whether the local node has permission to apply the model update; and updating, by at least one evaluation module, the current detection model with the model update.
According to another aspect of the invention, there is provided a computer-implemented method in a data processing system, the data processing system comprising a processor and a memory, the memory comprising instructions executable by the processor to cause the processor to implement a system for updating a detection model, the method comprising: updating at least one local node, comprising: receiving, by a monitoring module of at least one local node, a model update; determining, by a diagnostic module of the local node, the current monitoring model being used by the local node; determining, by an evaluation module of the local node, whether the model update should be applied to the current monitoring model; determining, by the evaluation module of the local node, whether the local node has permission to apply the model update; and updating, by the evaluation module of the local node, the current detection model with the model update.
According to another aspect of the invention, there is provided a computer program product for updating a detection model, the computer program product comprising at least one computer-readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to update at least one local node by: receiving, by at least one monitoring module, the model update; determining, by at least one diagnostic module, a current detection model; determining, by at least one evaluation module, whether the model update should be applied to the current detection model; determining, by at least one evaluation module, whether the local node has permission to apply the model update; and updating, by at least one evaluation module, the current detection model with the model update.
According to yet another aspect of the invention, there is provided a computer program product for creating and updating a detection model, the computer program product comprising at least one computer-readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to: creating a model update, the model update comprising: detecting, by a diagnostic module, a significant change in the system data; determining, by the diagnostic module, a list of all current detection models involved in detecting the significant change; analyzing, by the diagnostic module, the system data involved in detecting the significant change; generating, by the diagnostic module, the model update, wherein the generated model update does not include any system data; and transmitting, by the monitoring module, the model update; distributing the model update, including: receiving, by a central module, the model update; analyzing, by the central module, a database of available models; determining, by the central module, a priority for the model update; determining, by the central module, which local nodes should receive the model update; and transmitting, by the central module, the model update to at least one local node; and updating at least one local node, including: receiving, by at least one monitoring module, the model update; determining, by at least one diagnostic module, a current detection model; determining, by at least one evaluation module, whether the model update should be applied to the current detection model; determining, by at least one evaluation module, whether the local node has permission to apply the model update; and updating, by at least one evaluation module, the current detection model with the model update.
According to yet another aspect of the invention, there is provided a computer program product for creating and updating a detection model, the computer program product comprising at least one computer-readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to: creating a model update comprising: detecting, by a first diagnostic module, a significant change in the system data; determining, by the first diagnostic module, a list of all current detection models involved in detecting the significant change; analyzing, by the first diagnostic module, system data involved in detecting the significant change; generating, by the first diagnostic module, the model update, wherein the generated model update does not include any system data; and transmitting, by the first monitoring module, the model update; and updating at least the second local node, including: receiving, by at least a second monitoring module, the model update; determining, by at least a second diagnostic module, the current detection model; determining, by at least a second evaluation module, whether the model update should be applied to the current detection model; determining, by at least the second evaluation module, whether the local node has permission to apply the model update; and updating, by at least the second evaluation module, the current detection model with the model update.
Additional features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying drawings.
Drawings
The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. For the purpose of illustrating the disclosure, there is shown in the drawings embodiments which are presently preferred, it being understood, however, that the disclosure is not limited to the specific embodiments disclosed.
FIG. 1 depicts a block diagram of an exemplary update system including a single local node;
FIG. 2 depicts a block diagram of an exemplary update system including a plurality of local nodes connected by a central module;
FIG. 3 depicts a block diagram of an exemplary update system including a plurality of directly connected local nodes;
FIG. 4 illustrates a flow chart of an exemplary method for updating an analytics system using an update system and a manually created model update;
FIG. 5 depicts a flow diagram of an exemplary method for updating an analytics system using a model update created by one of the local nodes; wherein the local nodes are connected by a central module;
FIG. 6 depicts a flowchart of an exemplary method of updating an analytics system using an updated model created by one of the local nodes, wherein the local nodes are directionally connected; and
FIG. 7 depicts a block diagram of an example data processing system in which aspects of the illustrative embodiments may be implemented.
Detailed Description
The description and claims may utilize the terms "a," "an," "at least one," and "one or more" with respect to particular features and elements of the illustrative embodiments. It should be understood that these terms and phrases are intended to state that at least one particular feature or element is present in a particular illustrative embodiment, but may also be present in more than one. That is, these terms/phrases are not intended to limit the description or claims to a single feature/element that is present or to require the presence of multiple such features/elements. Rather, these terms/phrases only require at least a single feature/element, where a plurality of such features/elements are possible within the scope of the specification and claims.
Moreover, it should be appreciated that the following description uses a number of different examples of various elements of the illustrative embodiments to further illustrate exemplary implementations of the illustrative embodiments and to facilitate an understanding of the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and not exhaustive of the various possibilities for implementing the mechanisms of the illustrative embodiments. In view of this description, it will be apparent to those of ordinary skill in the art that many other alternative implementations of these different elements may be utilized in addition to or in place of the examples provided herein without departing from the scope of the present disclosure.
The present disclosure may be systems, methods, and/or computer program products. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.
The computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a head disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device such as a punch card, or a protruding structure in a slot having instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses traveling through a fiber optic cable), or an electrical signal transmitted through an electrical wire.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing/processing device, or to an external computer or external storage device, via a network, such as the internet, a Local Area Network (LAN), a Wide Area Network (WAN), and/or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing/processing device.
The computer-readable program instructions for performing the operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as JavaTMSmalltalk, C + +, etc.), and conventional procedural programming languages (e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, an electronic circuit comprising, for example, a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), can execute computer-readable program instructions by personalizing the electronic circuit with state information of the computer-readable program instructions in order to perform aspects of the present disclosure.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein comprise an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative embodiments, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
By way of overview, a cognitive system is a special purpose computer system or collection of computer systems configured with hardware and/or software logic (in combination with hardware logic on which software executes) to simulate human cognitive functions. These cognitive systems apply humanoid characteristics to convey and manipulate ideas that, when combined with the inherent strength of digital computing, can solve problems with high accuracy and flexibility on a large scale. IBM WatsonTMIs an example of one such cognitive system that is capable of processing human-readable language and recognizing inferences between text passages with human-like accuracy at a much faster rate and on a much larger scale than humans. Generally, such cognitive systems are capable of performing the following functions:
navigating complexity of human language and understanding
Ingesting and processing large amounts of structured and unstructured data
Generating and evaluating hypotheses
Weighing and evaluating responses based solely on relevant evidence
Providing specific consultations, insights and guidance
Improving knowledge through machine learning process and learning through each iteration and interaction
Enabling decisions at points of influence (context guidance)
Scaling in proportion to tasks
Extending and augmenting human expertise and cognition
Identifying resonances, human attributes and features of natural language
Inferring language-specific or agnostic properties from natural language
From highly relevant collections of data points (images, text, speech, memory and recall)
Experience-based situational awareness prediction and awareness for human cognition
The questions are answered based on natural language and specific evidence.
Embodiments herein relate to a system for updating an analytical model across a plurality of local nodes. As used herein, a separate "local node" refers to software installed by an end user (such as an individual or company). In some embodiments, the local node comprises a computer system. In some embodiments, the local nodes comprise a plurality of computer systems or servers controlled by end users. In some embodiments, each local node in the system uses a set of current analytical models that are specific to that local node. In some embodiments, each local node in the system accesses and analyzes system data generated by one or more analytical models. This system data is specific to each local node and may include sensitive or confidential information.
As used herein, an "analytical model" alone or a "model" alone is a software algorithm designed to detect certain events using data analysis techniques. In some embodiments, the analytical model detects data anomalies. In some embodiments, the analytics model detects fraud events. In some embodiments, the data analysis techniques used by the analytical model include, but are not limited to, data preprocessing techniques, calculation of one or more statistical parameters, statistical ratios based on classification or groups, calculation of probabilities, classification techniques such as data clustering and data matching, regression analysis, and gap analysis. In some embodiments, the software of the local node includes one or more analytical models. In some embodiments, the software of the local node includes one or more analytical models and deterministic rules. In some embodiments, the software of the local node includes one or more analytical models for fraud detection. In some embodiments, the software of the local node includes one or more analytical models for regulatory compliance or non-compliance. In some embodiments, the software of the local node includes one or more models and deterministic rules for fraud detection. In some embodiments, the software of the local node includes one or more models for regulatory compliance or non-compliance and deterministic rules.
In some embodiments, the update system receives one or more model updates and pushes the updates to the applicable local nodes. In some embodiments, the update system pushes updates to all local nodes in the system. In some embodiments, the update system pushes updates only to the selected local nodes. In some embodiments, the update system determines which local nodes receive the model update push.
In some embodiments, each individual local node receiving a model update checks the update against the current model of the analysis system, and if applicable, the update system will update the current model. In some embodiments, the update system receives one or more manually created model updates. In some embodiments, the update system receives one or more model updates created by a local node of the update system. In some embodiments, the local nodes of the update system are connected by a central hub or module that is not itself a local node. In some embodiments, the local modes of the update system are directly connected to each other, e.g., as a decentralized network.
In some embodiments, the update system, including any local nodes, is a stand-alone system that creates and pushes model updates for any software system that uses the analytical model. In some embodiments, the update system itself is a component or subsystem of a larger analysis system (e.g., an analysis system for fraud detection).
FIG. 1 depicts a block diagram representation of the components, outputs, and data flow of an exemplary single local node of an update system 100. The local node comprises three main modules or subsystems: a monitoring module 101, a diagnostic module 102, and an evaluation module 103.
The monitoring module 101 monitors one or more factors to determine whether a model update process is required. In some embodiments, the monitoring module 101 checks the time since the last update process and initiates an update process if sufficient time has elapsed. In some embodiments, the monitoring module 101 initiates the update process if 6 hours, 12 hours, 1 day, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, 10 days, 15 days, 30 days, 1 month, 2 months, 3 months, 6 months, or 1 year has elapsed since the last update process. In some embodiments, the monitoring module 101 initiates the update process upon receiving a model update pushed from a source external to the local node 100. For example, the monitoring module 101 may receive model updates pushed from a central module of the update system, another local node, or directly from an update system administrator.
In some embodiments, the monitoring module 101 may initiate the update process when signaled by the diagnostic module 102. In some embodiments, the diagnostic module 102 analyzes the system data 104 and may signal the monitoring module 101 to initiate an update process if one or more data thresholds have been met. For example, if the analysis of the system data 104 by the diagnostic module indicates that the event detection increases above the data threshold or that the event detection decreases below the data threshold, the diagnostic module 102 may signal the monitoring module 101. In some embodiments, the data threshold may be set manually, for example, by an end user. In some embodiments, the data threshold may be determined automatically by the diagnostic module 102, for example, if the event detection rate increases by a significant value over a moving average detection rate over a week.
When an update process is initiated, the monitoring module 101 will query for available model updates. In some embodiments, the monitoring module 101 will query a central module of the update system, another local mode, or an update system administrator. In some embodiments, if the update process is initiated by a model update pushed from a source external to local node 100, monitoring module 101 will not query for additional available model updates. In some embodiments, if the update process is initiated by a model update pushed from a source external to local node 100, monitoring module 101 will still query for additional available model updates.
When the monitoring module 101 has completed all available queries and has received at least one model update, the monitoring module 101 passes the model update to the diagnostic module 102. The diagnostic module 102 compares the model update to a database of current models 105 available in the local node. In some embodiments, the diagnostic module 102 will classify model updates for the current models 105 regardless of whether those current models 105 are actively used. In some embodiments, the diagnostic module 102 classifies model updates to the system data 104 generated by the current model 105 of the application activity.
When the diagnostic module 102 has received a model update and at least compared the model update to the database of current models 105, the diagnostic module 102 will pass the model update and all available comparison and other analysis data to the evaluation module 103. The evaluation module 103 updates the evaluation model to determine if the update 106 should be applied. In some embodiments, the evaluation module 103 will automatically apply the model update, changing or modifying the current model 105 with the model update. In some embodiments, the evaluation module 103 updates the analytical model to determine whether such model already exists in the database of the current model 105. In some embodiments, the evaluation module 103 will run a model update on the relevant system data 104 or the relevant classified data generated by the diagnostic module 102 to determine whether the model update will provide the local node 100 with system data that is different from the system data that the current model 105 can generate. In some embodiments, the evaluation module 103 will not automatically apply any updates or perform any analysis unless authorized by the end user or administrator of the local node 100.
Fig. 2 depicts a block diagram of presentation components, outputs and data flow of an update system having a plurality of local nodes 200. The update system comprises a central module 201 connecting all local nodes in the update system 200. Fig. 2 depicts two local nodes generally categorized as 210 and 220. In some embodiments, there is no limit to the number of local nodes that may be present in the update system 200. It should be understood that local nodes 210 and 220 are generally the same as the local nodes described in FIG. 1, and each include a monitoring module 211, 221, a diagnostic module 212, 222, and an evaluation module 213, 223. Each local node further includes its own database of system data 214, 224 and current models 215, 225. It should be understood that each local node may have different system data and current models. In some embodiments, system data 214 may or may not be the same or similar to system data 224. In some embodiments, the current model 215 may or may not be the same or similar to the current model 225.
The central module 201 does not exist in any local node but in a separate location, such as a centralized management server. In some embodiments, the central module 201 may send and receive information from the monitoring modules 211, 221. In some embodiments, the central module 201 may send and receive information from any monitoring module in the update system. The central module 201 may access a master database that updates the available models 202 of the system. The database of available models 202 is a list of all possible analytical models currently existing in the update system. In some embodiments, a database of current models (e.g., current model 215) in an individual node is equivalent to the dataset of available models 202. In some embodiments, the database of current models (e.g., current model 215) in an individual node is not identical to the dataset of available models 202, but contains at least one model in common with the database of available models 202.
In some embodiments, when a monitoring module (e.g., monitoring module 211) in an individual node initiates a query for available model updates, the monitoring module will be in electronic communication with central module 201.
In some embodiments, each individual node may communicate with one or more end users. For example, in FIG. 2, evaluation module 213 of node 210 may communicate with end user 217. In some embodiments, any module of a separate node may communicate with an end user. In some embodiments, a separate node communicates with the end user to provide information about the update process to the end user. In some embodiments, a separate node communicates with the end user to provide information to the end user regarding the results of the update, such as which models are updated. In some embodiments, a separate node communicates with the end user to request authorization of the end user before updating any model.
Fig. 3 depicts another block diagram representation of the components, outputs and data flow of an update system having a plurality of local nodes 300. The update system 300 depicts two local nodes, generally classified as 310 and 320. In some embodiments, there is no limit to the number of local nodes that may be present in the update system 300. It should be understood that the local nodes 310 and 320 are substantially the same as the local nodes described in fig. 1 and 2, each including a monitoring module 311, 321, a diagnostic module 312, 322 and an evaluation module 313, 323. Each local node further includes its own database of system data 314, 324 and current models 315, 325. It should be understood that each local node may have different system data and current models. In some embodiments, system data 314 may or may not be the same or similar to system data 324. In some embodiments, the current model 315 may or may not be the same or similar to the current model 325.
Unlike fig. 2, the update system 300 does not have any type of central module connecting all local nodes. Instead, each local node is directly connected to each other via the network. In some embodiments, each monitoring module is in electronic communication with each other monitoring module in the update system 300. For example, as shown in FIG. 3, the monitoring module 311 is in electronic communication with the monitoring module 321.
In some embodiments, when an update process has been initiated in a separate node, the monitoring module of that node will query another local node in the update system 300. For example, when an update process is initiated in the local node 310, the monitoring module 311 will query the monitoring module 321 of the local node 320. In some embodiments, when an update process has been initiated in a separate node, the monitoring module of that node will query all other local nodes in the update system 300. In some embodiments, when an update process has been initiated in a separate node, the monitoring module of that node will only query selected other nodes in the update system 300. In some embodiments, when an update process is initiated in a separate node, the monitoring module of that node will only query one other node in the update system 300.
In any of the embodiments herein, a system administrator may create an updated model and manually add it to the update system. For example, a system administrator may create an updated model and submit the model to the central module 201, as shown in FIG. 2. As another example, a system administrator may create an updated model and submit the model to the monitoring module 321, as shown in FIG. 3. In some embodiments, the update process may be initiated in all nodes of all nodes in the update system when the updated model has been added to any of the update systems described herein.
In any embodiment herein, any local node of the update system may initiate a model update and automatically push it to the rest of the update system. In some embodiments, it is advantageous for the local node to generate its own model updates, as this allows the update system to respond quickly to increases in fraud detection without the involvement of the end user or administrator. For example, the diagnostic module 212 shown in FIG. 2 analyzes the system data 214 and detects an increase in fraud detection greater than a preset threshold. The diagnostic module 212 continues to list one or more models for detecting increased fraud and analyzes the system data 214 to determine key features and conditions of the linkage between the one or more models and the data. The diagnostic module 212 then strips the model of any particular data to the system data 214 and the local node 210. The monitoring module 211 then sends the model to the central module 201, and the central module 201 will then determine whether the model is suitable for use as a model update for the update system 200.
In any embodiment where a local node initiates a model update of an update system, it is important that the local node's specific system data is not shared with any central hub or other local nodes in the update system. In some embodiments, the diagnostic module that creates the model to be shared with the update system creates a new model that is independent of any particular system data from the local node. In some embodiments, the new model includes one or more of: one or more algorithms, creation date and time, number of events detected over a given time period, metadata or advanced aggregation statistics (such as total transaction time values), and one or more threshold points for triggering updates. In some embodiments, the new model includes ratio statistics of one or more data set means. In some embodiments, the new model may detect deviations from the ratio statistics of one or more data set averages to determine future positive results. In some embodiments, the new model includes one or more network or image graphs representing one or more models. In some embodiments, the new model includes one or more network or image graphs representing the new model.
In any of the embodiments herein, the components of the update system may be stored in the same location as, for example, software installed in an internal server system at a company (such as a bank). In some embodiments, some components of any update system disclosed herein are stored in different locations, such as part of a cloud-based service.
FIG. 4 depicts a flow diagram of an exemplary method for pushing model updates using manually created models in an update system having a central module 400. In some embodiments, the method 400 may be used with the update system depicted in FIG. 2. First, a system administrator manually creates a new model to be used to update the system 401. In some embodiments, the new model includes a new or updated algorithm or algorithms. In some embodiments, the new model includes information about the criteria necessary for use of the new model, such as the type of traffic, the amount of system data required, or the type of detection performed by the model. In some embodiments, the new model includes priority information about how critical the model is to the update system. For example, it must be deduced that a new model to all local nodes will be given the highest possible priority. In some embodiments, the priority information is classified as low priority, medium priority, or high priority.
Next, the new model created by the system administrator is pushed to the update system that receives model 402. In some embodiments, a central module of the update system receives the model. Upon receipt of the model 402, the central module then updates the model database 403. For example, the central module 201 will update a database of available models 202 in the update system 200 depicted in FIG. 2.
The update system will then determine the applicable end-users for the new model 404. In some embodiments, the central module determines which end users are eligible. In some embodiments, in addition to the priority information of the new model, the central module determines to which end users the model update applies by comparing the standard information in the new model with information about each end user. For example, if the model updates for credit card fraud detection are of medium priority, the central module will identify which local nodes in the update system are involved in credit card fraud detection, and then push the model 405 out to those identified local nodes. Model updates will not be pushed out to any remaining local nodes, however, when each of those remaining local nodes initiates the update process, the local node may receive the update, for example, if sufficient time has passed to trigger the monitoring module without the update. In another example, if the model update for credit card fraud detection has a high priority, the central module will output the model 405 to all local nodes. In another example, if the model update for credit card fraud detection has a low priority, the central module does not push the model out to any local node immediately, but waits for each local node to initiate the update process itself.
Once the model update has been sent out from the central module, it is received 411 by at least one local node. In some embodiments, model updates are received simultaneously by multiple local nodes. In some embodiments, the model update is received by a monitoring module in any of the embodiments described herein.
In some embodiments, once the local node receives the model update 411, it is not automatically installed. First, the local node will consult the current model database to see if the model update will replace any existing models 412. The local node will then determine the relevance of the model update to the node 413. For example, in the local node 210 of the update system 200 depicted in FIG. 2, model updates are received by the monitoring module 211 and then passed to the diagnostic module 212. The diagnostic module 212 first consults the current model 215 database and then determines the relevance of the model updates to the local node 210. In some embodiments, the diagnostic module 212 will end the update process after the determine correlation step 413. In some embodiments, if the diagnostic module 212 determines that no model update is required for the local node, the diagnostic module 212 will end the update process after the determine relevance step 413. In some embodiments, if the diagnostic module 212 determines that a model update already exists in the local node, the diagnostic module 212 will end the update process after the determine relevance step 413. In some embodiments, if the model update carries a high priority, the diagnostic module 212 will automatically bypass the determine relevance step 413.
In some embodiments, once the local node has determined that a model update will be relevant or necessary, the local node will determine whether it has permission 414 to apply the model update. In some embodiments, the evaluation module of the local node determines whether the local node has permission to apply the model update. In some embodiments, the local node will not have permission to install the model update. In some embodiments, the local node will not have automatic permission to install any model updates. In some embodiments, the local node must consult or query for permission from the end user before installing the model update 416. For example, once the diagnostic module 212 has determined that the model update is relevant or the model update has a sufficiently high priority to bypass the determine relevance step 413, the model update is passed to the evaluation module 213. The evaluation module 213 then checks the updated permission settings of the local node. In some embodiments, if the assessment module 213 determines that it does not have permission to install the model update, the assessment module 213 will end the update process. In some embodiments, prior to installing the model update, the assessment module 213 will consult the end-user, for example by issuing a user prompt or by sending an email or other communication to the end-user.
Once the local node determines that it has permission to do so 415, the local node will install the model update. In some embodiments, the assessment module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models to the current model database in the local node. In some embodiments, the model update replaces one or more models in the current model database in the local node. For example, after the permissions have been established, the assessment module 213 updates the current model 215 database with model updates.
In some embodiments, once update 415 is complete, the local node creates an output report 417. In some embodiments, the output report is shared with the end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report contains information about the model update, including, for example, the type of model being updated, whether any old models were replaced, the date and time of the update, whether the new model is currently valid, or any combination thereof.
FIG. 5 depicts a flow diagram of an exemplary method of pushing model updates in an update system having a central module, where the model updates are automatically created from a local node in the update system 500. In some embodiments, the method 500 may be used with the update system depicted in FIG. 2. First, a local node in the update system will detect a change in the results from its existing model 501. In some embodiments, the local node in the update system will detect a change in the fraud detection rate. In some embodiments, the change in the fraud detection rate is an increase in fraud detection greater than a preset threshold. In some embodiments, the change in fraud detection rate is a significant increase or decrease in fraud over a given period of time. In some embodiments, the local node in the update system will detect changes in the magnitude of fraud detected. In some embodiments, the change in the magnitude of fraud is an increase in the value or dollar amount of the detected fraud event by more than a preset threshold. In some embodiments, the change in the magnitude of fraud is a significant increase in the value or dollar amount of detected fraudulent events compared to a moving average or number of detected events. For example, the diagnostic module 212 as depicted in fig. 2 analyzes the system data 214 and detects an increase in the fraud detection rate that is greater than the standard deviation from the 3 month moving average fraud detection rate.
Once a change in results from their existing models has been detected 501, the local node will list all models involved in the detection 502. In some embodiments, the local node will list all the models involved in directly generating the event detected in step 501. In some embodiments, the local node will list all models involved in directly and indirectly generating the event detected in step 501. In some embodiments, when an event is detected in step 501, the local node will list all the actively running models. For example, by accessing both the system data 214 and the current models 215 database, the diagnostic module 212 will list all the algorithmic models involved in directly and indirectly generating the fraud event previously detected in step 501.
Once the local node has listed the model 502 related to the detected change 501, the local node will analyze the data 503 involved in generating the event that caused the detected change. In some embodiments, the local node analyzes the system data to determine characteristics and conditions associated with the model listed in step 502 in the event detected in the generating step 501. In some embodiments, local node analysis may include, but is not limited to, ordinary least squares, penalty regression, generalized additive models, quantile regression, logistic regression, and gated linear models. In some embodiments, the local node analysis will be a transformation variant of one or more relevant models that reduces the complexity of those models. For example, monotonicity constraints are applied to a nonlinear, non-monotonic model to orient the model around variable relationships known to be true, or the utilization of monotonic neural networks for machine learning applications. In some embodiments, the relevant visualization will be a relevant but less complex model that approximates the applicable model or models, particularly machine learning models. For example, surrogate models, locally interpretable model-agnostic interpretation (LIME), maximum activation analysis, linear regression, and sensitivity analysis.
Once the local node has listed the model 502 and analyzed the relevant data 503, the local node may generate features 504 of the model update that will be sent to the rest of the update system. In some embodiments, the diagnostic module of the local node generates model features 504. In some embodiments, the characteristics of the model update are local node agnostic, i.e., the model update can be used by any local node in the update system. Thus, the model update generated by the local node is stripped of any particular data of the local node. In some embodiments, the model update features include one or more of: one or more algorithms, creation date and time, number of events detected over a given time period, metadata or advanced aggregation statistics (such as total transaction time values), and one or more threshold points for triggering updates. In some embodiments, the model update feature comprises a ratio statistic of one or more data set means. In some embodiments, the model update may detect a deviation from the ratio statistics of one or more data set averages to determine future positive results. In some embodiments, the model update features include one or more network or image graphs representing one or more models. In some embodiments, the model update features include one or more network or image graphs representing the new model.
Once the local node has generated the model features 504, the local node may output model updates 505. In some embodiments, the local node will output the model update to a central module of the update system that receives the model update 506. For example, the monitoring module 211 of the local node 210 may receive model updates from the diagnostic module 212, and then the monitoring module 211 may send the model updates to the central module 201 that received the model updates.
Upon receiving the model update 506 from the local node, the central module of the update system will then consult the model database 507. In some embodiments, the central module consults the model database to determine if a model update already exists. In some embodiments, the central module consults the model database to determine whether the model update replaces an existing model in the database or a new model of the database. In some embodiments, when the central module consults the model database and determines that a model update may replace or modify an existing model in the database, the central module may pull model information about the existing model. In some embodiments, the model information may include one or more of the following: the model creation date and time, the date and time the model was last updated, and how many local nodes are currently using the model. For example, upon receiving a model update from local node 210, central module 201 checks the model update against the database of available models 202. The central module 201 determines if a model update already exists in the database of available models 202 and if so, the central module 201 will pull relevant information about any existing models.
After the central module of the update system consults the model database 507, the central module will then determine 508 the priority of the model update. In some embodiments, the priority of the model update will be listed as high, medium, or low. In some embodiments, the priority levels of model updates will be listed in numerical scale, such as between the range of 1 to 10 or other common numerical ranges. In some embodiments, the central module determines the priority of the model update by comparing the model update characteristics to a predetermined scale. In some embodiments, the central module determines the priority of the model update by comparing the model update characteristics to a model database. In some embodiments, the central module determines the priority of the model update by comparing the model update characteristics to model information stored in an existing model database. In some embodiments, the comparison of the model update characteristics to the existing model information results in a priority level that is then translated into a priority level.
For example, after the central module 201 of the update system 200 checks the model updates against the available model database 202 for credit card fraud detection, the central module 201 determines that a similar model already exists in the database and pulls information about the existing model. The central module 201 then compares the model update characteristics with the existing model information and calculates the priority levels. As a first example, the central module determines that the existing model of credit card fraud detection has not been updated within a year, the model update is a direct replacement for the existing model, and the model update may increase the performance of detecting credit card fraud over a range of usage conditions. These differences result in a high priority level which the central module 201 changes to. As a second example, the central module determines that an existing model for credit card fraud detection has been recently updated, and would only expect model updates to improve the performance of detecting credit card fraud with a sufficiently large user base that only a few known end users have. These differences result in a relatively low priority level which the central module 201 changes to a medium priority.
After determining the priorities, the update system will then determine the end users 509 to which the new model applies. In some embodiments, the central module determines which end users are eligible. In some embodiments, in addition to the priority information of the new model, the central module determines to which end users the model update is applicable by comparing the model update characteristics with information about each end user. For example, if the model updates for credit card fraud detection are of medium priority, the central module will identify which local nodes in the update system are involved in credit card fraud detection, and then push the model 510 out to those identified local nodes. Model updates will not be pushed out to any remaining local nodes, however when each of those remaining local nodes initiates the update process, the local node may receive the update, for example if sufficient time has passed to trigger the monitoring module without the update. In another example, if the model update for credit card fraud detection has a high priority, the central module will output the model 510 to all local nodes. In another example, if the model update for credit card fraud detection has a low priority, the central module does not push the model out to any local node immediately, but waits for each local node to initiate the update process itself.
Once the model update has been sent out from the central module, it is received 511 by at least one local node. In some embodiments, model updates are received simultaneously by multiple local nodes. In some embodiments, the model update is received by a monitoring module in any of the embodiments described herein.
In some embodiments, the model update 511 is not automatically installed once it is received by the local node. First, the local node will consult the current model database to see if the model update will replace any existing models 512. The local node will then determine the relevance of the model update to the node 513. For example, in the local node 210 of the update system 200 depicted in FIG. 2, model updates are received by the monitoring module 211 and then passed to the diagnostic module 212. The diagnostic module 212 first consults the current model 215 database and then determines the relevance of the model updates to the local node 210. In some embodiments, the diagnostic module 212 will end the update process after the determine correlation step 513. In some embodiments, if the diagnostic module 212 determines that no model update is required for the local node, the diagnostic module 212 will end the update process after the determine relevance step 513. In some embodiments, if the diagnostic module 212 determines that a model update already exists in the local node, the diagnostic module 212 will end the update process after the determine correlation step 513. In some embodiments, if the model update carries a high priority, the diagnostic module 212 will automatically bypass the determine relevance step 513.
In some embodiments, once the local node has determined that a model update will be relevant or necessary, the local node will determine whether it has permission 514 to apply the model update. In some embodiments, the evaluation module of the local node determines whether the local node has permission to apply the model update. In some embodiments, the local node will not have permission to install the model update. In some embodiments, the local node will not have automatic permission to install any model updates. In some embodiments, the local node must consult or query for permission from the end user before installing the model update 516. For example, once the diagnostic module 212 has determined that the model update is relevant or the model update has a sufficiently high priority to bypass the determined relevance step 513, the model update is passed to the evaluation module 213. The evaluation module 213 then checks the updated permission settings of the local node. In some embodiments, if the assessment module 213 determines that it does not have permission to install the model update, the assessment module 213 will end the update process. In some embodiments, prior to installing the model update, the assessment module 213 will consult the end-user, for example by issuing a user prompt or by sending an email or other communication to the end-user.
Once the local node determines 515 that it has permission to do so, the local node will install the model update. In some embodiments, the assessment module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models to the current model database in the local node. In some embodiments, the model update replaces one or more models in the current model database in the local node. For example, after the permissions have been established, the assessment module 213 updates the current model 215 database with model updates.
In some embodiments, once the update 515 is complete, the local node creates an output report 517. In some embodiments, the output report is shared with the end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report contains information about the model update, including, for example, the type of model being updated, whether any old models were replaced, the date and time of the update, whether the new model is currently valid, or any combination thereof.
FIG. 6 depicts a flowchart of an exemplary method of pushing model updates in an update system without a central module, where the model updates are automatically created from local nodes in the update system 600. In some embodiments, the method 600 may be used with the update system depicted in FIG. 3. First, a local node in the update system will detect a change in the results from its existing model 601. In some embodiments, the local node in the update system will detect a change in the fraud detection rate. In some embodiments, the change in the fraud detection rate is an increase in fraud detection greater than a preset threshold. In some embodiments, the change in fraud detection rate is a significant increase or decrease in fraud over a given period of time. In some embodiments, the local node in the update system will detect changes in the magnitude of fraud detected. In some embodiments, the change in the magnitude of fraud is an increase in the value or dollar amount of the detected fraud event by more than a preset threshold. In some embodiments, the change in the magnitude of fraud is a significant increase in the value or dollar amount of detected fraudulent events compared to a moving average or number of detected events. For example, the diagnostic module 312 as depicted in fig. 3 analyzes the system data 314 and detects an increase in the fraud detection rate that is greater than the standard deviation of the 3 month moving average fraud detection rate.
Once a change 601 in the results from its existing model has been detected, the local node will list all models 602 involved in the detection. In some embodiments, the local node will list all models that are directly involved in generating the event detected in step 601. In some embodiments, the local node will list all models that are directly and indirectly involved in generating the event detected in step 601. In some embodiments, when an event 601 is detected in step, the local node will list all the actively running models. For example, by accessing both the system data 314 and the current model database 315, the diagnostic module 312 will list all of the algorithmic models directly and indirectly involved in generating the previously detected fraudulent event in step 601.
Once the local node has listed the model 602 associated with the detected change 601, the local node will analyze the data 603 involved in generating the event that caused the detected change. In some embodiments, the local node analyzes the system data to determine characteristics and conditions associated with the model listed in step 602 when generating the event detected in step 601. In some embodiments, local node analysis may include, but is not limited to, ordinary least squares, penalty regression, generalized additive models, quantile regression, logistic regression, and gated linear models. In some embodiments, the local node analysis will be a transformation variant of one or more relevant models that reduces the complexity of those models. For example, monotonicity constraints are applied to a nonlinear, non-monotonic model to orient the model around variable relationships known to be true, or the utilization of monotonic neural networks for machine learning applications. In some embodiments, the relevant visualization will be a relevant but less complex model that approximates the applicable model or models, particularly machine learning models. For example, surrogate models, locally interpretable model-agnostic interpretation (LIME), maximum activation analysis, linear regression, and sensitivity analysis.
Once the local node has listed the models 602 and analyzed the relevant data 603, the local node may generate features 604 of the model update that will be sent to the rest of the update system. In some embodiments, the diagnostic module of the local node generates model features 604. In some embodiments, the characteristics of the model update are local node agnostic, i.e., the model update can be used by any local node in the update system. Thus, the model update generated by the local node is stripped of any particular data of the local node. In some embodiments, the model update features include one or more of: one or more algorithms, creation date and time, number of events detected over a given time period, metadata or advanced aggregation statistics (such as total transaction time values), and one or more threshold points for triggering updates. In some embodiments, the model update feature comprises a ratio statistic of one or more data set means. In some embodiments, the model update may detect a deviation from the ratio statistics of one or more data set averages to determine future positive results. In some embodiments, the model update features include one or more network or image graphs representing one or more models. In some embodiments, the model update features include one or more network or image graphs representing the new model.
Once the local node has generated the model features 604, the local node may output a model update 605. In some embodiments, the local node will output the model update to at least one other local node of the update system, which receives the model update 611. In some embodiments, the local node will output the model update to all other local nodes of the update system. For example, the monitoring module 311 of the local node 310 may receive the model update from the diagnostic module 312, and then the monitoring module 311 may send the model update to another local node 320 that received the model update.
In some embodiments, once the local node receives the model update 611, it is not automatically installed. First, the local node will consult the current model database to see if the model update will replace any existing models 612. The local node will then determine the relevance of the model update to the node 613. For example, in the local node 310 of the update system 300 depicted in fig. 3, model updates are received by the monitoring module 311 and then passed to the diagnostic module 312. The diagnostic module 312 first queries the current model database 315 and then determines the relevance of the model update to the local node 310. In some embodiments, the diagnostic module 312 will end the update process after the determine relevance step 613. In some embodiments, if the diagnostic module 312 determines that the local node does not require a model update, the diagnostic module 312 will end the update process after the determine relevance step 613. In some embodiments, if the diagnostic module 312 determines that a model update already exists in the local node, the diagnostic module 312 will end the update process after the determine relevance step 613.
In some embodiments, once the local node has determined that a model update will be relevant or necessary, the local node will determine if it has permission 614 to apply the model update. In some embodiments, the evaluation module of the local node determines whether the local node has permission to apply the model update. In some embodiments, the local node will not have permission to install the model update. In some embodiments, the local node will not have automatic permission to install any model updates. In some embodiments, the local node must consult or query the permissions 616 from the end-user before installing the model update. For example, once the diagnostic module 312 has determined that the model update is relevant, the model update is passed to the evaluation module 313. The evaluation module 313 then checks the updated permission settings of the local node. In some embodiments, if the evaluation module 313 determines that it does not have permission to install the model update, the evaluation module 313 will end the update process. In some embodiments, prior to installing the model update, the evaluation module 313 will consult the end-user, for example by issuing a user prompt or by sending an email or other communication to the end-user.
Once the local node determines 615 that it has permission to do so, the local node will install the model update. In some embodiments, the assessment module installs the model update. In some embodiments, any module of the update system installs the model update. In some embodiments, the model update installs one or more new models to the current model database in the local node. In some embodiments, the model update replaces one or more models in the current model database in the local node. For example, after the permissions have been established, the assessment model 313 updates the current model database 315 with model updates.
In some embodiments, once the update 615 is complete, the local node creates an output report 617. In some embodiments, the output report is shared with the end user. In some embodiments, the output report is shared with a central module of the update system. In some embodiments, the output report contains information about the model update, including, for example, the type of model being updated, whether any old models were replaced, the date and time of the update, whether the new model is currently valid, or any combination thereof.
In some embodiments, the users of any of the systems disclosed herein may be one or more human users, such as the known "human-in-the-loop" system. In some embodiments, a user of any of the systems disclosed herein can be a computer system, artificial intelligence ("AI"), cognitive or non-cognitive algorithms, and the like.
The following table is a non-exclusive and non-exhaustive list of examples using any of the systems and methods disclosed herein:
Figure BDA0003552839200000141
FIG. 7 depicts a block diagram of an example data processing system 700 in which aspects of the illustrative embodiments are implemented. Data processing system 700 is an example of a computer, such as a server or a client, in which computer usable code or instructions implementing the processes for any of the illustrative embodiments described herein may be located. In some embodiments, fig. 7 represents a server computing device, such as a server, that implements a system for interpreting analysis results described herein.
In the depicted example, data processing system 700 may employ a hub architecture including a north bridge and memory controller hub (NB/MCH)701 and a south bridge and input/output (I/O) controller hub (SB/ICH) 702. Processing unit 703, main memory 704, and graphics processor 705 may be connected to NB/MCH 701. Graphics processor 705 may be connected to NB/MCH 701 through an Accelerated Graphics Port (AGP). Network adapter 706 connects to SB/ICH 702. Audio adapter 707, keyboard and mouse adapter 708, modem 709, Read Only Memory (ROM)710, Hard Disk Drive (HDD)711, compact disk drive (CD or DVD)712, Universal Serial Bus (USB) ports and other communications ports 713, and PCI/PCIe devices 714 may be connected to SB/ICH 702 through bus system 716. PCI/PCIe devices 714 may include Ethernet adapters, add-in cards, and PC cards for notebook computers. ROM 710 may be, for example, a flash basic input/output system (BIOS). The HDD 711 and the optical drive 712 may use an Integrated Drive Electronics (IDE) or Serial Advanced Technology Attachment (SATA) interface. A super I/O (SIO) device 715 may be connected to SB/ICH 702.
An operating system may run on the processing unit 703. An operating system may coordinate and provide control of various components within data processing system 700. As a client, the operating system may be a commercially available operating system. Object-oriented programming system (such as Java)TMA programming system) may run in conjunction with the operating system and provide calls to the operating system from object-oriented programs or applications executing on data processing system 700. As a server, data processing system 700 may be running a high-level mutual execution operating system or
Figure BDA0003552839200000151
Of operating systems
Figure BDA0003552839200000152
Data processing system 700 may be a Symmetric Multiprocessor (SMP) system including a plurality of processors in processing unit 703. Alternatively, a single processor system may be employed.
Instructions for the operating system, the object-oriented programming system, and applications or programs are located on storage devices, such as HDD 711, and are loaded into main memory 704 for execution by processing unit 703. The processes of an embodiment of the medical record error detection system may be performed by processing unit 703 using computer usable program code, which may be located in a memory such as, for example, main memory 704, ROM 710, or one or more peripheral devices.
The bus system 716 may include one or more buses. The bus system 716 may be implemented using any type of communication fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture. A communication unit, such as modem 709 or network adapter 706, may include one or more devices that may be used to transmit and receive data.
Those of ordinary skill in the art will appreciate that the hardware required to operate any of the systems and methods described herein may vary depending on the embodiment. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives and the like, may be used in addition to or in place of the hardware depicted. Further, any of the systems described herein may take the form of any of a number of different data processing systems, including but not limited to client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants, and the like. Essentially, any system described herein may be any known or later developed data processing system without architectural limitation.
The systems and methods of the drawings are not exclusive. Other systems and processes may be derived in accordance with the principles of the embodiments described herein to achieve the same objectives. It is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may occur to those skilled in the art without departing from the scope of the present embodiments. As described herein, the various systems, subsystems, agents, managers and processes may be implemented using hardware components, software components and/or combinations thereof.
Although the present invention has been described with reference to exemplary embodiments, the present invention is not limited thereto. Those skilled in the art will appreciate that many variations and modifications may be made to the preferred embodiments of the present invention. It is, therefore, intended that the appended claims be interpreted as covering all such equivalent variations as fall within the scope of the invention.

Claims (35)

1. A computer program product for updating a detection model, the computer program product comprising at least one computer-readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to update at least one local node by:
receiving, by at least one monitoring module, the model update;
determining, by at least one diagnostic module, the current detection model;
determining, by at least one evaluation module, whether the model update should be applied to the current detection model;
determining, by at least one evaluation module, whether the local node has permission to apply the model update; and
updating, by the at least one evaluation module, the current detection model with the model update.
2. The computer program product of claim 1, wherein program instructions executable by the processor further cause the processor to distribute the model update to the at least one local node.
3. The computer program product of claim 2, wherein the model update is distributed to at least two local nodes.
4. The computer program product of claim 2, wherein the model update is distributed to the at least one local node by a central module.
5. The computer program product of claim 4, wherein the central module distributes the model updates to the at least one local node by:
receiving, by the central module, the model update;
analyzing, by the central module, a database of available models;
determining, by the central module, a priority for the model update;
determining, by the central module, which local nodes should receive the model update; and
transmitting, by the central module, the model update to the at least one local node.
6. The computer program product of claim 5, wherein the database of available models comprises models available to all local nodes.
7. The computer program product of claim 5, wherein the priority is determined by comparing features of the model update to model information in an existing model database.
8. The computer program product of claim 2, wherein the model update is distributed to the at least one local node by another local node.
9. The computer program product of claim 1, wherein program instructions executable by the processor further cause the processor to create a model update.
10. The computer program product of claim 9, wherein the creation of a model update occurs in a local node.
11. The computer program product of claim 10, wherein the creation of the model update is performed by:
detecting, by the diagnostic module, a significant change in system data;
determining, by the diagnostic module, a list of all current detection models involved in detecting the significant change;
analyzing, by the diagnostic module, the system data involved in detecting the significant change;
generating, by the diagnostic module, the model update; and
transmitting, by the monitoring module, the model update.
12. The computer program product of claim 11, wherein the generated model update comprises one or more elements selected from the group consisting of: one or more algorithms, a creation date and time, a number of events detected within a given time period, aggregate statistics, and one or more threshold points for triggering the model update.
13. The computer program product of claim 11, wherein the generated model update does not include any system data specific to the local node that generated the model update.
14. The computer program product of claim 11, wherein the monitoring module transmits the model update to a central module external to the local node.
15. The computer program product of claim 11, wherein the monitoring module transmits the model update to the monitoring module from another local node.
16. A system for updating a detection model, comprising:
at least one local node comprising:
a monitoring module;
a diagnostic module;
an evaluation module;
one or more current detection models; and
system data generated by the current inspection model; and
a memory comprising instructions for execution by at least one processor, the instructions configured to:
receiving, by the monitoring module, a model update;
determining, by the diagnostic module, the current detection model;
determining, by the evaluation module, whether the model update should be applied to the current detection model;
determining, by the evaluation module, whether the local node has permission to apply the model update; and
updating, by the evaluation module, the current detection model with the model update.
17. The system of claim 16, wherein the system further comprises at least two local nodes.
18. The system of claim 17, wherein the system further comprises a central module, and wherein the monitoring module of each local node is in electronic communication with the central module.
19. The system of claim 18, wherein the system further comprises a database of all available models of the system in electronic communication with the central module.
20. The system of claim 18, wherein the monitoring module is configured to receive a model update.
21. The system of claim 20, wherein the monitoring module is capable of receiving model updates from a system administrator or from a local node.
22. The system of claim 17, wherein the instructions are further configured to:
detecting, by the diagnostic module, a significant change in system data;
determining, by the diagnostic module, a list of all current test models involved in the testing step;
analyzing, by the diagnostic module, the system data involved in the detecting step;
generating, by the diagnostic module, the model update;
transmitting, by the monitoring module, the model update.
23. The system of claim 22, wherein the generated model update comprises one or more elements selected from the group consisting of: one or more algorithms, a creation date and time, a number of events detected within a given time period, aggregate statistics, and one or more threshold points for triggering the model update.
24. The system of claim 22, wherein the generated model update does not include any system data specific to the local node that generated the model update.
25. The system of claim 16, further comprising:
at least a second local node;
a central module, wherein the monitoring module of each local node is in electronic communication with the central module; and
a database of all available models of the system in electronic communication with the central module; and
a memory comprising instructions for execution by at least one processor, the instructions configured to:
creating a model update comprising:
detecting, by the diagnostic module, a significant change in system data;
determining, by the diagnostic module, a list of all current test models involved in the testing step;
analyzing, by the diagnostic module, the system data involved in the detecting step;
generating, by the diagnostic module, the model update;
transmitting, by the monitoring module, the model update;
distributing model updates, comprising:
receiving, by the central module, the model update;
analyzing, by the central module, a database of available models;
determining, by the central module, a priority for the model update;
determining, by the central module, which local nodes should receive the model update; and
transmitting, by the central module, the model update; and
updating at least one local node, comprising:
receiving, by at least one monitoring module, a model update;
determining, by at least one diagnostic module, the current detection model;
determining, by at least one evaluation module, whether the model update should be applied to the current detection model;
determining, by at least one evaluation module, whether the local node has permission to apply the model update; and
updating, by the at least one evaluation module, the current detection model with the model update.
26. A computer-implemented method in a data processing system comprising a processor and a memory, the memory including instructions executable by the processor to cause the processor to implement a system to update a detection model, the method comprising:
updating at least one local node, comprising:
receiving, by a monitoring module of at least one local node, a model update;
a diagnostic module of the local node determining a current detection model being used by the local node;
determining, by an evaluation module of the local node, whether the model update should be applied to the current detection model;
determining, by the evaluation module of the local node, whether the local node has permission to apply the model update; and
updating, by the evaluation module of the local node, the current detection model with the model update.
27. The method of claim 26, wherein the method further comprises updating at least two local nodes.
28. The method of claim 27, wherein the method further comprises distributing the model updates by a central module.
29. The method of claim 28, wherein the method further comprises analyzing, by the central module, a database of available models.
30. The method of claim 28, wherein the method further comprises determining, by the central module, a priority of the model update.
31. The method of claim 28, wherein the method further comprises determining, by the central module, which local nodes should receive the model update.
32. The method of claim 27, the method further comprising:
creating a model update comprising:
detecting, by the diagnostic module, a significant change in system data;
determining, by the diagnostic module, a list of all current test models involved in the testing step;
analyzing, by the diagnostic module, the system data involved in the detecting step;
generating, by the diagnostic module, the model update;
transmitting, by the monitoring module, the model update.
33. The method of claim 32, wherein the generated model update comprises one or more elements selected from the group consisting of: one or more algorithms, a creation date and time, a number of events detected within a given time period, aggregate statistics, and one or more threshold points for triggering the model update.
34. The method of claim 32, wherein the generated model update does not include any system data specific to the local node that generated the model update.
35. The method of claim 26, further comprising:
creating a model update comprising:
detecting, by the diagnostic module, a significant change in system data;
determining, by the diagnostic module, a list of all current detection models involved in detecting the significant change;
analyzing, by the diagnostic module, the system data involved in detecting the significant change;
generating, by the diagnostic module, the model update; and
transmitting, by the monitoring module, the model update;
distributing the model update, including:
receiving, by a central module, the model update;
analyzing, by the central module, the database of available models;
determining, by the central module, a priority for the model update;
determining, by the central module, which local nodes should receive the model update; and
transmitting, by the central module, the model update to at least one local node; and
updating at least one local node, comprising:
receiving, by at least one monitoring module, the model update;
determining, by at least one diagnostic module, the current detection model;
determining, by at least one evaluation module, whether the model update should be applied to the current detection model;
determining, by at least one evaluation module, whether the local node has permission to apply the model update; and
updating, by the at least one evaluation module, the current detection model with the model update.
CN202080065674.3A 2019-09-20 2020-09-15 Updating detection models and maintaining data privacy Pending CN114424164A (en)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US16/577,770 US11216268B2 (en) 2019-09-20 2019-09-20 Systems and methods for updating detection models and maintaining data privacy
US16/577,770 2019-09-20
US16/577,774 US11188320B2 (en) 2019-09-20 2019-09-20 Systems and methods for updating detection models and maintaining data privacy
US16/577,774 2019-09-20
PCT/IB2020/058564 WO2021053509A1 (en) 2019-09-20 2020-09-15 Updating detection models and maintaining data privacy

Publications (1)

Publication Number Publication Date
CN114424164A true CN114424164A (en) 2022-04-29

Family

ID=74884596

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202080065674.3A Pending CN114424164A (en) 2019-09-20 2020-09-15 Updating detection models and maintaining data privacy

Country Status (5)

Country Link
JP (1) JP2022548945A (en)
CN (1) CN114424164A (en)
DE (1) DE112020003693T5 (en)
GB (1) GB2603686A (en)
WO (1) WO2021053509A1 (en)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8407160B2 (en) * 2006-11-15 2013-03-26 The Trustees Of Columbia University In The City Of New York Systems, methods, and media for generating sanitized data, sanitizing anomaly detection models, and/or generating sanitized anomaly detection models
CN106610854B (en) * 2015-10-26 2020-02-18 阿里巴巴集团控股有限公司 Model updating method and device
CN107229966B (en) * 2016-03-25 2021-03-16 阿里巴巴集团控股有限公司 Model data updating method, device and system
CN108921301B (en) * 2018-06-29 2020-06-02 长扬科技（北京）有限公司 Self-learning-based machine learning model updating method and system

Also Published As

Publication number Publication date
GB202205042D0 (en) 2022-05-18
WO2021053509A1 (en) 2021-03-25
JP2022548945A (en) 2022-11-22
DE112020003693T5 (en) 2022-04-21
GB2603686A (en) 2022-08-10

Similar Documents

Publication Publication Date Title
KR102008707B1 (en) Risk management system
US11176206B2 (en) Incremental generation of models with dynamic clustering
US11790256B2 (en) Analyzing test result failures using artificial intelligence models
CA3154647C (en) Maintaining data privacy in a shared detection model system
US11216268B2 (en) Systems and methods for updating detection models and maintaining data privacy
US20190362417A1 (en) Systems and methods for interpreting analytical results
CN114730399A (en) Distributed analysis alert model degradation based on usage risk tolerance rating
US20180012181A1 (en) Method of collaborative software development
US20220291966A1 (en) Systems and methods for process mining using unsupervised learning and for automating orchestration of workflows
Lyu et al. Towards a consistent interpretation of aiops models
US20220318681A1 (en) System and method for scalable, interactive, collaborative topic identification and tracking
CN114402301B (en) System and method for maintaining data privacy in a shared detection model system
Chouchen et al. Learning to predict code review completion time in modern code review
US20210142233A1 (en) Systems and methods for process mining using unsupervised learning
US20220083320A1 (en) Maintenance of computing devices
CN114424164A (en) Updating detection models and maintaining data privacy
US11188320B2 (en) Systems and methods for updating detection models and maintaining data privacy
US20210397545A1 (en) Method and System for Crowdsourced Proactive Testing of Log Classification Models
US20210397544A1 (en) Crowdsourced Proactive Testing System for Named Entity Recognition Models in IT Support
US11755950B2 (en) Methods for refining data set to represent output of an artificial intelligence model
US20230229492A1 (en) Automated context based data subset processing prioritization
US11270258B2 (en) Dynamic inventory segmentation with predicted prediction time window
Maneriker et al. Auditing Fairness Online through Interactive Refinement
WO2023215515A1 (en) Computer-implemented methods, systems comprising computer-readable media, and electronic devices for narrative representation of a network computing environment
WO2023220256A1 (en) Computer-implemented methods, systems comprising computer-readable media, and electronic devices for enabled intervention into a network computing environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination