Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
  • H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/30—Semantic analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/451—Execution arrangements for user interfaces
  • G06F9/453—Help systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
  • G06N20/20—Ensemble learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/004—Artificial life, i.e. computing arrangements simulating life
  • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N7/00—Computing arrangements based on specific mathematical models
  • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L63/00—Network architectures or network communication protocols for network security
  • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
  • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
  • H04L63/1416—Event detection, e.g. attack signature detection
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L63/00—Network architectures or network communication protocols for network security
  • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
  • H04L63/1433—Vulnerability analysis
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L63/00—Network architectures or network communication protocols for network security
  • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
  • H04L63/1441—Countermeasures against malicious traffic
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L63/00—Network architectures or network communication protocols for network security
  • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
  • H04L63/1441—Countermeasures against malicious traffic
  • H04L63/1491—Countermeasures against malicious traffic using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/02—Details
  • H04L12/16—Arrangements for providing special services to substations
  • H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
  • H04L12/1813—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast for computer conferences, e.g. chat rooms
  • H04L12/1818—Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/06—Management of faults, events, alarms or notifications
  • H04L41/0681—Configuration of triggering conditions

Definitions

• a model-based agent can handle partially observable environments. Its current state is stored inside the agent maintaining some kind of structure which describes the part of the world which cannot be seen. This knowledge about "how the world works” is called a model of the world, hence the name "model-based agent.”
• a model-based reflex agent should maintain some sort of internal model that depends on the percept history and thereby reflects at least some of the unobserved aspects of the current state. Percept history and impact of action on the environment can be determined by using internal model. It then chooses an action in the same way as reflex agent.
• Virtual agents are increasingly being deployed in enterprises to handle interactions with customers or with employees. As these virtual agents are taking on more functions in enterprises, they are increasingly becoming a target for attack (for example, spam, extraction, poisoning, and evasion attacks). Therefore, there is a need in the art to address the aforementioned problem.
• Figure 2 is a flowchart depicting operational steps of a program for detecting and mitigating adversarial conversations with virtual agents, in accordance with an embodiment of the present invention.
• Figure 3 illustrates an example of a program for detecting and mitigating adversarial conversations with virtual agents, in accordance with an embodiment of the present invention.
• Computing device 110 may be a desktop computer, a laptop computer, a tablet computer, a specialized computer server, a smartphone, a wearable device (e.g., smart watch, personal fitness device, personal safety device), or any programmable computer system known in the art with an interactive display or any other computer system known in the art.
• computing device 110 represents a computer system utilizing clustered computers and components that act as a single pool of seamless resources when accessed through network 185, as is common in data centers and with cloud computing applications.
• computing device 110 is representative of any programmable electronic device or combination of programmable electronic devices capable of executing machine-readable program instructions and communicating with other computer devices via a network.
• An I/O device interfacing with graphical user interface 130 may be connected to computing device 110, which may operate utilizing wired (e.g., USB port) or wireless network communications (e.g., infrared, NFC, etc.).
• Computing device 110 may include components, as depicted and described in further detail with respect to Figure 4, in accordance with embodiments of the present invention.
• Storage 160 (e.g., a database) located on computing device 110, represents any type of storage device capable of storing data that is accessed and utilized by computing device 110. In other embodiments, storage 160 represents multiple storage devices within computing device 110. Storage 160 stores information such as, but not limited to, account information, credentials for authentication, user preferences, lists of preferred users, previously visited websites, history of visited Wi-Fi portals, and the history of the location of the computing device.
• Handshaking facilitates connecting heterogeneous computing systems, or equipment, over a communication channel without the need for user intervention to set parameters.
• server 120 initiates the handshake process by sending a massage to computing device 110 indicating that server 120 wants to establish a communication channel in order to gain access to programs on computing device 110.
• bot shield database 182 stores information from a Markov detection model.
• bot shield database 182 stores historical information, flagged as inappropriate, of unlikely interactions between a virtual agent and a user.
• Bot shield database 182 stores the history of the transition frequencies from the dialog states between a user and a virtual agent.
• Deception engine 190 is a sub program of program 200 that automatically adjusts the fidelity of a response, to a user, by virtual agent program 175 to deter a potential attack. Deception engine 190 mitigates attacks by changing the fidelity, or exactness, of model responses without altering original dialog flow with a user. Deception engine 190 changes the exactness of the model responses given to a user by selecting a fidelity level of response according to the current user risk scores. The higher the user risk, the lower the precision of the model response given to the high-risk user.
• deception engine 190 in response to high-risk activity by a user, changes the fidelity of the response consistent with the risk level of the user.
• deception engine 190 changes the fidelity of the model response to "please reenter your credit card number to confirm.”
• Deception engine 190 is triggered if the user risk score passes a certain predefined threshold. In an embodiment, deception engine 190 triggers mitigation actions based on the specific risk level of a user and the fact that a specific threshold was passed based on the user responses to virtual agent program 175.
• deception engine 190 redirects the user to a honeypot model.
• deception engine 190 uses a model that mimics the functionality of the original model, but is trained with data that is loosely representative of the original baseline truth, but is similar enough to fool an attacker. Deceptive responses by deception engine 190 can help invalidate the already extracted information in the attacker.
• deception engine 190 changes the fidelity of model responses without altering the original dialogue flow with a user.
• deception engine 190 slows or disrupts the information accumulation in a hypothetical adversary.
• deception engine 190 escalates the conversation to a human responder.
• program 200 operates as a code snippet within one or more applications on computing device 110.
• Code snippets define the scope of interactivity between the snippets and the application, (e.g., program 200 hosted by a web browser application on server 120).
• program 200 is a function within web browser 150, and the processes of program 200 occur automatically (i.e., without user intervention) during operation of web browser 150 as initiated by program 200.
• the dynamic code snippet elements provide scripting support.
• the variables enable dialog between program 200, through server 120, graphical user interface 130, web browser 150, and virtual agent program 175.
• program 200 is capable of being implemented as an independent anomaly detection system that is capable of interfacing with the dialogue system of virtual agent program 175 to provide conversation security.
• Program 200 detects anomalous and suspicious conversations by leveraging conversational context through preceding model queries by a conversation.
• program 200 can operate as a plug-in for virtual agent as a monitoring capability operating on conversation logs.
• the anomaly detection subsystem could be used as a stand-alone program to feed an operation dashboard with anomaly monitoring results.
• deception engine 190 and probe 195 are integrated with or cooperate with the dialogue runtime as deception engine 190 and probe 195 are manipulating conversation flow with a user.
• Each subsystem is extensible and able to learn from the encounters with various attackers. Extensible means that additional detection models are capable of being implemented to program 200, additional mitigation responses are capable of being added to program 200, and additional probe choices are capable of being added to program 200.
• program 200 computes a risk value for a user utilizing the Markov detection module individually.
• program 200 computes a high-risk value for a user because the dialog log entry to the virtual agent is characteristic of known attacks.
• the dialog log entry by the user conforms to the pattern of a known attack and is deemed intrusive.
• Program 200 uses the similarity of the log entry to known attacks to compute a high-risk value to the user.
• program 200 merges the risk scores from the individual detection models into a single risk score, R, using a weighted ensemble function. The weights of the function may be adapted over time.
• Program 200 updates the user risk score in the virtual agent using the merged risk value.
• Program 200 incrementally updates all the anomaly detection models in the ensemble using the dialogue log entry.
• program 200 receives a dialog log entry and utilizes a combination of two anomaly detection models to compute a risk score for a user.
• Program 200 utilizes the timing anomaly detection model to determine that the response time by the user is consistent with the timing pattern associated with a known attacker.
• Program 200 assigns an individual risk score, "r1” based on the timing anomaly detection model.
• Program 200 utilizes the dialog progression anomaly detection model to determine that the dialog log entry to the virtual agent are consistent with a dialog progression pattern of a known attacker.
• Program 200 assigns an individual risk score, "r2” based on the dialog progression anomaly model.
• Program 200 assesses an "R" score that exceeds the threshold as determined by the institution in this example. Domain models 320 was trained on sensitive data, and contains confidential information. Program 200 also assigns an "R” score to adversary 315 based upon the adversarial inputs, as determined by program 200, in response to virtual nurse 375.
• step 230 program 200 permits access. More specifically, in response to determining that the risk score does not pass a threshold (decision step 220, "no" branch), program 200 permits access to virtual agent program 175 (step 230). In this example, program 200 determines that the risk value, as determined in step 210, does not meet a threshold value "R.” In an example, program 200 analyzes each utterance by a user to virtual agent program 175 and determine that the responses are by a human. Program 200 accesses bot shield database 182 and views a history of similar utterances that were determined to be human with high confidence. Based upon the exactness of the utterances and the consistency of the responses with a history of acceptable responses, program 200 allows a user access to virtual program 175.
• Program 200 activates a mitigation actions that terminate the conversation by stating "I am not trained on this- for further help please call 1-800"
• Program 200 intervenes each time the user risk score passes the "high risk” threshold, based upon the "R" value assigned in step 210.
• Program 200 changes the dialogue flow and redirects virtual nurse 375's response to a previously generated low fidelity response to mitigate the interaction between adversary 315 and virtual nurse 375.
• program 200 delays the response to a user in accordance with the user's response to the probe or the calculated risk score.
• a user's utterances to virtual nurse 375 become increasingly similar with known patterns of high-risk utterances.
• Program 200 increases the risk score of the user proportionally to each high-risk response. As the risk score gets higher, program 200 introduces a longer delay before sending a response back to the user.
• Fidelity level 1 is the highest level and is consistent with the original response by virtual agent 175 to a human user with a low risk score, fidelity level 2 would be lower, fidelity level 3 is even lower, up to fidelity level N, as determined by institutional or user preferences.
• program 200 is capable of generating lower fidelity responses through creating additional models that result in lower fidelity responses to a perceived attacker.
• Program 200 is capable of using a progressive model dilution. Diluting a model is a method that uses a previously trained model as the ground truth for the diluted model. As a result, program 200, through deception engine 190, makes the previously trained model a less accurate version of the baseline truth in the original model.
• Program 200 is capable of infinitely chaining each low-fidelity response to a progressively lower fidelity response.
• program 200 responsive to determining that the risk value passes a threshold (decision step 220, "yes" branch), program 200 performs mitigating actions in response to a spam attack.
• program 200 determines, based on the signature response to a threshold high fidelity question presented by virtual nurse 375.
• Program 200 assists virtual nurse to validate that adversary 315 is an attacker by providing increasingly lower fidelity questions to adversary 315.
• Program 200 presents adversary 315 with a fidelity level 3 question.
• Program 200 determines that adversary 315 is bot generated spam traffic attempting to drive up operations cost for the virtual agent based on the incomprehensible responses to the questions presented by program 200.
• program 200 uses probe 195 to quickly develop a risk score for a user or to further analyze an indeterminate user and assign a risk score as a function of the further analysis by probe 195.
• Program 200 ranks available probes by plausibility to occur in a normal conversation.
• Program 200 injects occasional probes based on the current risk score of a user and program 200 can adjust the strength and need for a probe, and the frequency of injection based on the current risk score of a user.
• Program 200 evaluates the response by a user to a probe and updates the risk score accordingly, or program 200 can employ further probes.
• Program 200 adds information to bot shield database 182 as a function of the information provided by probe 195.
• Program 200 through probe 195, sends out a request or other form of inquiry to get more information back from the user in response to the probe.
• Program 200 through probe 195, is capable of intervening in a conversation between a user and virtual nurse 375 to further assess a risk score as assigned to the user.
• program 200 interjects into a conversation and directly requests that the user to prove that the user is a human through one or more probes, such as a "Captchas.”
• program 200 employs probe 195 in response to a new user, or a user with no signature history of responses stored in bot shield database 182. Probe 195 may be engaged at a low frequency by program 200.
• program 200 employs a probe to verbal utterances by a user and inserts phrases such as, "I'm sorry I am not trained on this, could you please rephrase," "Did you mean X?" (where X is something that virtual agent program 175 is highly confident that X is not what the previous user utterance was related to, i.e., negative confirmation), or program 200 can utilize probe 195 to employ a superfluous question that requires more than a "yes” or "no” answer and that relates to the current context, (e.g., "when did you first acquire your car” in a conversation about car insurance).
• program 200 responsive to determining that the risk value passes a threshold (decision step 220, "yes" branch), performs mitigating actions in response to a poisoning attack.
• virtual nurse 375 continuously learns through production use. The more interaction that virtual nurse 375 has with "good” users the better that virtual nurse 375 functions and the more virtual nurse 375 evolves.
• program 200 determines, based on the utterances by adversary 315, that adversary 315 is altering virtual nurse 375's training data. Program 200 determines that the topics of conversation imitated, and continued, by adversary 315 are driving the conversation, originally resigned to health care topics, off course into unrelated topic areas.
• I/O interface(s) 412 allows for input and output of data with other devices that may be connected to server 120.
• I/O interface 412 may provide a connection to external devices 418 such as a keyboard, keypad, a touch screen, and/or some other suitable input device.
• External devices 418 can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.
• Software and data used to practice embodiments of the present invention, e.g., program 200 can be stored on such portable computer readable storage media and can be loaded onto persistent storage 408 via I/O interface(s) 412.
• I/O interface(s) 412 also connect to a display 420.
• Display 420 provides a mechanism to display data to a user and may be, for example, a computer monitor.
• the present invention may be a system, a method, and/or a computer program product.
• the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
• Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
• the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
• a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
• the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
• electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
• These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
• These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
• each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
• the functions noted in the block may occur out of the order noted in the figures.
• two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Security & Cryptography (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computing Systems (AREA)
• Software Systems (AREA)
• General Physics & Mathematics (AREA)
• Signal Processing (AREA)
• Computer Networks & Wireless Communication (AREA)
• Artificial Intelligence (AREA)
• Computer Hardware Design (AREA)
• Data Mining & Analysis (AREA)
• Evolutionary Computation (AREA)
• Mathematical Physics (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Computational Linguistics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Medical Informatics (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Mathematical Analysis (AREA)
• Pure & Applied Mathematics (AREA)
• Mathematical Optimization (AREA)
• Algebra (AREA)
• Probability & Statistics with Applications (AREA)
• Computational Mathematics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Molecular Biology (AREA)
• Human Computer Interaction (AREA)
• Computer And Data Communications (AREA)
• User Interface Of Digital Computer (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=66096155

Family Applications (1)

Country Status (5)

Cited By (3)

Families Citing this family (22)

Citations (3)

Family Cites Families (41)

• 2017
  • 2017-10-18 US US15/786,888 patent/US10574598B2/en not_active Expired - Fee Related
• 2018
  • 2018-10-10 CN CN201880067127.1A patent/CN111213161B/zh active Active
  • 2018-10-10 GB GB2007194.0A patent/GB2581741A/en not_active Withdrawn
  • 2018-10-10 WO PCT/IB2018/057830 patent/WO2019077440A1/en not_active Ceased
  • 2018-10-10 JP JP2020520029A patent/JP7083559B2/ja active Active
• 2020
  • 2020-01-20 US US16/747,007 patent/US11206228B2/en active Active

Patent Citations (3)

Also Published As

Similar Documents

Legal Events