WO2022060792A1 - Bot supervision - Google Patents
Bot supervision Download PDFInfo
- Publication number
- WO2022060792A1 WO2022060792A1 PCT/US2021/050401 US2021050401W WO2022060792A1 WO 2022060792 A1 WO2022060792 A1 WO 2022060792A1 US 2021050401 W US2021050401 W US 2021050401W WO 2022060792 A1 WO2022060792 A1 WO 2022060792A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- conversation
- prediction
- confidence
- communication
- bot
- Prior art date
Links
- 238000004891 communication Methods 0.000 claims abstract description 372
- 238000000034 method Methods 0.000 claims abstract description 94
- 230000015654 memory Effects 0.000 claims description 46
- 238000012544 monitoring process Methods 0.000 claims description 9
- 239000003795 chemical substances by application Substances 0.000 description 101
- 238000007726 management method Methods 0.000 description 94
- 230000003993 interaction Effects 0.000 description 26
- 230000004044 response Effects 0.000 description 26
- 230000008569 process Effects 0.000 description 24
- 238000010586 diagram Methods 0.000 description 21
- 238000010801 machine learning Methods 0.000 description 21
- 238000004422 calculation algorithm Methods 0.000 description 19
- 230000005540 biological transmission Effects 0.000 description 14
- 230000006870 function Effects 0.000 description 14
- 238000012545 processing Methods 0.000 description 14
- 238000004458 analytical method Methods 0.000 description 11
- 230000008859 change Effects 0.000 description 11
- 238000004590 computer program Methods 0.000 description 11
- 238000012549 training Methods 0.000 description 10
- 230000009471 action Effects 0.000 description 9
- 238000013507 mapping Methods 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 8
- 230000009466 transformation Effects 0.000 description 8
- 238000013500 data storage Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 7
- 230000005291 magnetic effect Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 230000000977 initiatory effect Effects 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 238000013478 data encryption standard Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 241000590419 Polygonia interrogationis Species 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 241001441724 Tetraodontidae Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000005022 packaging material Substances 0.000 description 1
- 238000012913 prioritisation Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/107—Computer-aided management of electronic mailing [e-mailing]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/216—Handling conversation history, e.g. grouping of messages in sessions or threads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
Definitions
- the present disclosure relates generally to hot operation and training. More specifically, techniques are provided to supervision of hot operation and training.
- bots may operate from a model. Although the model may be updated over time, bot errors, failures, or other operational breakdowns may nevertheless occur. As such, businesses may employ human agents to take over from such bot failures in order to meet customer needs and expectations. Requiring human agents to take over various tasks may be timeintensive, however, and the bots may continue to break down until the model is updated at some point in time. There is, therefore, a need in the art for improved methods and systems for supervising bot operation and training.
- the method may include storing a set of supervision criteria in memory that includes a threshold level of confidence, monitoring one or more messages in a conversation between a bot and a user of a client device, detecting that a bot prediction associated with the conversation fails to meet the threshold level of confidence, requesting input from an agent device that provides classification input regarding the conversation, routing the conversation to one or more endpoint devices in accordance with the classification input, updating a learning model based on the classification input, and providing for use of the updated learning model by the bot to make a next bot prediction.
- Certain embodiments of the present disclosure include a system.
- the system may include one or more data processors; and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform the methods described above and herein.
- Certain embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine -readable storage medium, including instructions configured to cause a data processing apparatus to perform the methods described above and herein.
- FIG. 1 shows a block diagram of an embodiment of a network interaction system
- FIG. 2 shows a block diagram of another embodiment of a network interaction system
- FIGS. 3A-3C show block diagrams of other embodiments of a network interaction system that includes a connection management system
- FIG. 4 shows a representation of a protocol-stack mapping of connection components’ operation
- FIG. 5 represents a multi-device communication exchange system according to an embodiment
- FIG. 6 shows a block diagram of an embodiment of a connection management system
- FIG. 7 shows a block diagram of a network environment for dynamically switching between bots and terminal devices during communication sessions
- FIG. 8 shows a block diagram representing a network environment for dynamically selecting endpoints across multiple channel environments
- FIG. 9 shows a block diagram representing a network environment for enhancing endpoint selection using machine-learning techniques.
- FIG. 10 shows an example process for supervising bot operation and training.
- FIG. 11 shows a block diagram representing a network environment in which a system for supervising bot operation and training may be implemented.
- FIG. 12 illustrates an exemplary interface for supervising bot operation and training.
- FIG. 1 shows a block diagram of an embodiment of a network interaction system 100 which implements and supports certain embodiments and features described herein. Certain embodiments relate to establishing a connection channel between a network device 105 (which can be operated by a user 110) and a terminal device 115 (which can be operated by an agent 120).
- the network interaction system 100 can include a client device 130 associated with a client 125.
- a user 110 can be an individual browsing a web site or accessing an online service provided by a remote server 140.
- a client 125 can be an entity that provides, operates, or runs the web site or the online service, or individuals employed by or assigned by such an entity to perform the tasks available to a client 125 as described herein.
- the agent 120 can be an individual, such as a support agent tasked with providing support or information to the user 110 regarding the website or online service. Out of a large number of agents, a subset of agents may be appropriate for providing support or information for a particular client 125.
- the agent 120 may be affiliated or not affiliated with the client 125. Each agent can be associated with one or more clients 125.
- a user 110 can be an individual shopping an online store from a personal computing device, a client 125 can be a company that sells products online, and an agent 120 can be a representative employed by the company. In various embodiments, the user 110, client 125, and agent 120 can be other individuals or entities.
- FIG. 1 shows only a single network device 105, terminal device 115 and client device 130
- an interaction system 100 can include multiple or many (e.g., tens, hundreds or thousands) of each of one or more of these types of devices.
- FIG. 1 shows only a single user 110, agent 120 and client 125
- an interaction system 100 can include multiple or many of each of one or more of such entities.
- a remote server 140 may also be configured to receive and respond to select network-device communications.
- a connection management system 150 can facilitate strategic routing of communications.
- a communication can include a message with content (e.g., defined based on input from an entity, such as typed or spoken input).
- the communication can also include additional data, such as data about a transmitting device (e.g., an IP address, account identifier, device type and/or operating system); a destination address; an identifier of a client; an identifier of a webpage or webpage element (e.g., a webpage or webpage element being visited when the communication was generated or otherwise associated with the communication) or online history data; a time (e.g., time of day and/or date); and/or destination address. Other information can be included in the communication.
- connection management system 150 routes the entire communication to another device.
- connection management system 150 modifies the communication or generates a new communication (e.g., based on the initial communication).
- the new or modified communication can include the message (or processed version thereof), at least some (or all) of the additional data (e.g., about the transmitting device, webpage or online history and/or time) and/or other data identified by connection management system 150 (e.g., account data associated with a particular account identifier or device).
- the new or modified communication can include other information as well.
- connection management system 150 can first estimate to which client (if any) the communication corresponds.
- connection management system 150 can identify a terminal device 115 associated with the client for communication with network device 105.
- the identification can include evaluating a profile of each of a plurality of agents (or experts or delegates), each agent (e.g., agent 120) in the plurality of agents being associated with a terminal device (e.g., terminal device 115). The evaluation can relate to a content in a network-device message.
- the identification of the terminal device 115 can include a technique described, for example, in U.S. Application Number 12/725,799, filed on March 17, 2010, which is hereby incorporated by reference in its entirety for all purposes.
- connection management system 150 can determine whether any connection channels are established between network device 105 and a terminal device associated with the client (or remote server 140) and, if so, whether such channel is to be used to exchange a series of communications including the communication.
- connection management system 150 can establish a connection channel between the network device 105 and terminal device 115. In some instances, connection management system 150 can transmit a message to the selected terminal device 115. The message may request an acceptance of a proposed assignment to communicate with a network device 105 or identify that such an assignment has been generated.
- the message can include information about network device 105 (e.g., IP address, device type, and/or operating system), information about an associated user 110 (e.g., language spoken, duration of having interacted with client, skill level, sentiment, and/or topic preferences), a received communication, code (e.g., a clickable hyperlink) for generating and transmitting a communication to the network device 105, and/or an instruction to generate and transmit a communication to network device 105.
- information about network device 105 e.g., IP address, device type, and/or operating system
- information about an associated user 110 e.g., language spoken, duration of having interacted with client, skill level, sentiment, and/or topic preferences
- code e.g., a clickable hyperlink
- connection management system 150 can allow connection management system 150 to monitor the communication exchange and to detect issues (e.g., as defined based on rules) such as non-responsiveness of either device or extended latency. Further, such a configuration can facilitate selective or complete storage of communications, which may later be used, for example, to assess a quality of a communication exchange and/or to support learning to update or generate routing rules so as to promote particular postcommunication targets.
- connection management system 150 can monitor the communication exchange in real-time and perform automated actions (e.g., rule-based actions) based on the live communications. For example, when connection management system 150 determines that a communication relates to a particular item (e.g., product), connection management system 150 can automatically transmit an additional message to terminal device 115 containing additional information about the item (e.g., quantity of item available, links to support documents related to the item, or other information about the item or similar items).
- automated actions e.g., rule-based actions
- a designated terminal device 115 can communicate with network device 105 without relaying communications through connection management system 150.
- One or both devices 105, 115 may (or may not) report particular communication metrics or content to connection management system 150 to facilitate communication monitoring and/or data storage.
- connection management system 150 may route select communications to a remote server 140.
- Remote server 140 can be configured to provide information in a predetermined manner. For example, remote server 140 may access defined one or more text passages, voice recording and/or files to transmit in response to a communication. Remote server 140 may select a particular text passage, recording or file based on, for example, an analysis of a received communication (e.g., a semantic or mapping analysis).
- Routing and/or other determinations or processing performed at connection management system 150 can be performed based on rules and/or data at least partly defined by or provided by one or more client devices 130.
- client device 130 may transmit a communication that identifies a prioritization of agents, terminal-device types, and/or topic/skill matching.
- client device 130 may identify one or more weights to apply to various variables potentially impacting routing determinations (e.g., language compatibility, predicted response time, device type and capabilities, and/or terminal-device load balancing), ft will be appreciated that which terminal devices and/or agents are to be associated with a client may be dynamic.
- Communications from client device 130 and/or terminal devices 115 may provide information indicating that a given terminal device and/or agent is to be added or removed as one associated with a client.
- client device 130 can transmit a communication with IP address and an indication as to whether a terminal device with the address is to be added or removed from a list identifying client-associated terminal devices.
- Each communication can occur over one or more networks 170.
- Any combination of open or closed networks can be included in the one or more networks 170.
- suitable networks include the Internet, a personal area network, a local area network (LAN), a wide area network (WAN), or a wireless local area network (WLAN).
- LAN local area network
- WAN wide area network
- WLAN wireless local area network
- Other networks may be suitable as well.
- the one or more networks 170 can be incorporated entirely within or can include an intranet, an extranet, or a combination thereof.
- a network in the one or more networks 170 includes a short-range communication channel, such as a Bluetooth or a Bluetooth Low Energy channel.
- communications between two or more systems and/or devices can be achieved by a secure communications protocol, such as secure sockets layer (SSL) or transport layer security (TLS).
- SSL secure sockets layer
- TLS transport layer security
- data and/or transactional details may be encrypted based on any convenient, known, or to be developed manner, such as, but not limited to, Data Encryption Standard (DES), Triple DES, Rivest-Shamir-Adleman encryption (RSA), Blowfish encryption, Advanced Encryption Standard (AES), CAST-128, CAST-256, Decorrelated Fast Cipher (DFC), Tiny Encryption Algorithm (TEA), extended TEA (XTEA), Corrected Block TEA (XXTEA), and/or RC5, etc.
- DES Data Encryption Standard
- Triple DES Rivest-Shamir-Adleman encryption
- AES Advanced Encryption Standard
- CAST-128, CAST-256 CAST-128, CAST-256
- DFC Decorrelated Fast
- a network device 105, terminal device 115 and/or client device 130 can include, for example, a portable electronic device (e.g., a smart phone, tablet, laptop computer, or smart wearable device) or a non-portable electronic device (e.g., one or more desktop computers, smart appliances, servers, and/or processors).
- Connection management system 150 can be separately housed from network, terminal and client devices or may be part of one or more such devices (e.g., via installation of an application on a device).
- Remote server 140 may be separately housed from each device and connection management system 150 and/or may be part of another device or system. While each device, server and system in FIG. 1 is shown as a single device, it will be appreciated that multiple devices may instead be used.
- a set of network devices can be used to transmit various communications from a single user, or remote server 140 may include a server stack.
- a software agent or application may be installed on and/or executable on a depicted device, system or server.
- the software agent or application is configured such that various depicted elements can act in complementary manners.
- a software agent on a device can be configured to collect and transmit data about device usage to a separate connection management system, and a software application on the separate connection management system can be configured to receive and process the data.
- FIG. 2 shows a block diagram of another embodiment of a network interaction system 200.
- FIG. 2 illustrates a variety of components configured and arranged to enable a network device 205 to communicate with one or more terminal devices 215.
- the depicted instance includes nine terminal devices 215 included in three local-area networks 235.
- a communication from network device 205 includes destination data (e.g., a destination IP address) that at least partly or entirely indicates which terminal device is to receive the communication.
- Network interaction system 200 can include one or more inter-network connection components 240 and/or one or more intra-network connection components 255 that can process the destination data and facilitate appropriate routing.
- Each inter-network connection components 245 can be connected to a plurality of networks 235 and can have multiple network cards installed (e.g., each card connected to a different network).
- an inter-network connection component 245 can be connected to a wide-area network 270 (e.g., the Internet) and one or more local-area networks 235.
- the communication in order for a communication to be transmitted from network device 205 to any of the terminal devices, in the depicted system, the communication must be handled by multiple inter-network connection components 245.
- inter-network connection component 245 When an inter-network connection component 245 receives a communication (or a set of packets corresponding to the communication), inter-network connection component 245 can determine at least part of a route to pass the communication to a network associated with a destination.
- the route can be determined using, for example, a routing table (e.g., stored at the router), which can include one or more routes that are pre-defined, generated based on an incoming message (e.g., from another router or from another device) or learned.
- inter-network connection components 245 include a router 260 and a gateway 265.
- An inter-network connection component 245 e.g., gateway 265 may be configured to convert between network systems or protocols.
- gateway 265 may facilitate communication between Transmission Control Protocol/Intemet Protocol (TCP/IP) and Internetwork Packet Exchange/Sequenced Packet Exchange (IPX/SPX) devices.
- TCP/IP Transmission Control Protocol/Intemet Protocol
- IPX/SPX Internetwork Packet Exchange/Sequenced Packet Exchange
- intra-network routing can be performed via an intra-network connection component 255, such as a switch 280 or hub 285.
- Each intra-network connection component 255 can be connected to (e.g., wirelessly or wired, such as via an Ethernet cable) multiple terminal devices 215.
- Hub 285 can be configured to repeat all received communications to each device to which it is connected.
- Each terminal device can then evaluate each communication to determine whether the terminal device is the destination device or whether the communication is to be ignored.
- Switch 280 can be configured to selectively direct communications to only the destination terminal device.
- a local-area network 235 can be divided into multiple segments, each of which can be associated with independent firewalls, security rules and network protocols.
- An intra-network connection component 255 can be provided in each of one, more or all segments to facilitate intra-segment routing.
- a bridge 280 can be configured to route communications across segments 275.
- various components analyze destination data in the communications. For example, such data can indicate which network a communication is to be routed to, which device within a network a communication is to be routed to or which communications a terminal device is to process (versus ignore). However, in some instances, it is not immediately apparent which terminal device (or even which network) is to participate in a communication from a network device.
- a set of terminal devices may be configured so as to provide similar types of responsive communications.
- a query in a communication from a network device may be responded to in similar manners regardless to which network device the communication is routed. While this assumption may be true at a high level, various details pertaining to terminal devices can give rise to particular routings being advantageous as compared to others.
- terminal devices in the set may differ from each other with respect to (for example) which communication channels are supported, geographic and/or network proximity to a network device and/or characteristics of associated agents (e.g., knowledge bases, experience, languages spoken, availability, general personality or sentiment, etc.). Accordingly, select routings may facilitate faster responses that more accurately and/or completely respond to a network-device communication.
- a complication is that static routings mapping network devices to terminal devices may fail to account for variations in communication topics, channel types, agent availability, and so on.
- FIGS. 3A-3C show block diagrams of other embodiments of a network interaction system 300a-c that includes a connection management system.
- Each of the depicted systems 300a- c show only 2 local-area networks 235 for simplicity, though it can be appreciated that embodiments can be extended to expand the number of local-area networks.
- Each of systems 300a- c include a connection management system 350, which can identify which terminal device is to communicate with network device 205, can establish and manage (e.g., maintain or close) connection channels, can determine whether and when to re-route communications in an exchange, and so on.
- connection management system 350 can be configured to dynamically, and in real-time, evaluate communications, agent availability, capabilities of terminal devices or agents, and so on, to influence routing determinations.
- connection management system 350 is associated with each of network device 205 and a remote server 340 (e.g., connection management system 350a is associated with network device 205 and connection management system 350b is associated with remote server 340).
- connection management system 350a and/or connection management system 350b can be installed or stored as an application on each of network device 205 and remote server 340, respectively. Execution of the application(s) can facilitate, for example, a communication between network device 205 and remote server 340 to identify a terminal device 215 selected to participate in a communication exchange with network device 205. The identification can be made based on one or more factors disclosed herein (e.g., availability, matching between a communication’s topic/level of detail with agents’ or terminal devices’ knowledge bases, predicted latency, channel-type availability, and so on).
- a client device 330 can provide client data indicating how routing determinations are to be made.
- client data can include: indications as to how particular characteristics are to be weighted or matched or constraints or biases (e.g., pertaining to load balancing or predicted response latency).
- Client data can also include specifications related to when communication channels are to be established (or closed) or when communications are to be re-routed to a different network device.
- Client data can be used to define various client-specific rules, such as rules for communication routing and so on.
- Connection management system 350b executing on remote server 340 can monitor various metrics pertaining to terminal devices (e.g., pertaining to a given client), such as which communication channels are supported, geographic and/or network proximity to a network device, communication latency and/or stability with the terminal device, a type of the terminal device, a capability of the terminal device, whether the terminal device (or agent) has communicated with a given network device (or user) before and/or characteristics of associated agents (e.g., knowledge bases, experience, languages spoken, availability, general personality or sentiment, etc.). Accordingly, connection management system 350b may be enabled to select routings to facilitate faster responses that more accurately and/or completely respond to a network-device communication based on the metrics.
- a communication exchange between network device 205 and remote server 340 can facilitate early identification of a destination address.
- Network device 205 may then use the destination address to direct subsequent communications.
- network device 205 may send an initial communication to remote server 340 (e.g., via one or more inter-network connections and a wide-area network), and remote server 340 may identify one or more corresponding clients.
- Remote server 340 may then identify a set of terminal devices associated with the one or more corresponding clients and collect metrics for those terminal devices.
- the metrics can be evaluated (e.g., by remote server 340) so as to select a terminal device to involve in a communication exchange, and information pertaining to the terminal device (e.g., an IP address) can be sent to network device 205.
- remote server 340 may continuously or periodically collect and evaluate metrics for various terminal devices and store evaluation results in a data store.
- remote server 340 can access the stored evaluation results from the data store and select a terminal device to involve in the communication exchange based on the stored evaluation results.
- connection management system 350 can be configured to serve as a relay and/or destination address.
- a set of network devices 205 may transmit communications, each identifying connection management system 350 as a destination.
- Connection management system 350 can receive each communication and can concurrently monitor a set of terminal devices (e.g., so as to generate metrics for each terminal device). Based on the monitoring and a rule, connection management system 350 can identify a terminal device 215 to which it may relay each communication.
- terminal device communications may similarly be directed to a consistent destination (e.g., of connection management system 350) for further relaying, or terminal devices may begin communicating directly with corresponding network devices.
- connection management system 350 is directly connected to intra-network components (e.g., terminal devices, intra-network connections, or other).
- intra-network components e.g., terminal devices, intra-network connections, or other.
- connection management system 350 may be associated with a connection component (e.g., inter-network connection component 245 or intra-network connection component 255) such that an application corresponding to connection management system 350 (or part thereof) is installed on the component.
- the application may, for example, perform independently or by communicating with one or more similar or complementary applications (e.g., executing on one or more other components, network devices or remotes servers).
- FIG. 4 shows a representation of a protocol-stack mapping 400 of connection components' operation. More specifically, FIG. 4 identifies a layer of operation in an Open Systems Interaction (OSI) model that corresponds to various connection components.
- OSI Open Systems Interaction
- the OSI model can include multiple logical layers 402-414.
- the layers are arranged in an ordered stack, such that layers 402-412 each serve a higher level and layers 404-414 is each served by a lower layer.
- the OSI model includes a physical layer 402.
- Physical layer 402 can define parameters physical communication (e.g., electrical, optical, or electromagnetic).
- Physical layer 402 also defines connection management protocols, such as protocols to establish and close connections.
- Physical layer 402 can further define a flow-control protocol and a transmission mode.
- a link layer 404 can manage node-to-node communications.
- Link layer 404 can detect and correct errors (e.g., transmission errors in the physical layer 402) and manage access permissions.
- Link layer 404 can include a media access control (MAC) layer and logical link control (LLC) layer.
- MAC media access control
- LLC logical link control
- a network layer 406 can coordinate transferring data (e.g., of variable length) across nodes in a same network (e.g., as datagrams).
- Network layer 406 can convert a logical network address to a physical machine address.
- a transport layer 408 can manage transmission and receipt quality.
- Transport layer 408 can provide a protocol for transferring data, such as a Transmission Control Protocol (TCP).
- TCP Transmission Control Protocol
- Transport layer 408 can perform segmentation/desegmentation of data packets for transmission and can detect and account for transmission errors occurring in layers 402-406.
- a session layer 410 can initiate, maintain and terminate connections between local and remote applications. Sessions may be used as part of remote-procedure interactions.
- a presentation layer 412 can encrypt, decrypt and format data based on data types known to be accepted by an application or network layer.
- An application layer 414 can interact with software applications that control or manage communications. Via such applications, application layer 414 can (for example) identify destinations, local resource states or availability and/or communication content or formatting. Various layers 402-414 can perform other functions as available and applicable.
- Intra-network connection components 422, 424 are shown to operate in physical layer 402 and link layer 404. More specifically, a hub can operate in the physical layer, such that operations can be controlled with respect to receipts and transmissions of communications. Because hubs lack the ability to address communications or filter data, they possess little to no capability to operate in higher levels. Switches, meanwhile, can operate in link layer 404, as they are capable of filtering communication frames based on addresses (e.g., MAC addresses).
- addresses e.g., MAC addresses
- inter-network connection components 426, 428 are shown to operate on higher levels (e.g., layers 406-414).
- routers can filter communication data packets based on addresses (e.g., IP addresses). Routers can forward packets to particular ports based on the address, so as to direct the packets to an appropriate network.
- Gateways can operate at the network layer and above, perform similar filtering and directing and further translation of data (e.g., across protocols or architectures).
- connection management system 450 can interact with and/or operate on, in various embodiments, one, more, all or any of the various layers.
- connection management system 450 can interact with a hub so as to dynamically adjust which terminal devices the hub communicates.
- connection management system 450 can communicate with a bridge, switch, router or gateway so as to influence which terminal device the component selects as a destination (e.g., MAC, logical or physical) address.
- a connection management system 450 can monitor, control, or direct segmentation of data packets on transport layer 408, session duration on session layer 410, and/or encryption and/or compression on presentation layer 412.
- connection management system 450 can interact with various layers by exchanging communications with (e.g., sending commands to) equipment operating on a particular layer (e.g., a switch operating on link layer 404), by routing or modifying existing communications (e.g., between a network device and a terminal device) in a particular manner, and/or by generating new communications containing particular information (e.g., new destination addresses) based on the existing communication.
- a particular layer e.g., a switch operating on link layer 404
- connection management system 450 can influence communication routing and channel establishment (or maintenance or termination) via interaction with a variety of devices and/or via influencing operating at a variety of protocol-stack layers.
- FIG. 5 represents a multi-device communication exchange system 500 according to an embodiment.
- System 500 includes a network device 505 configured to communicate with a variety of types of terminal devices over a variety of types of communication channels.
- network device 505 can transmit a communication over a cellular network (e.g., via a base station 510).
- the communication can be routed to an operative network 515.
- Operative network 515 can include a connection management system 520 that receives the communication and identifies which terminal device is to respond to the communication. Such determination can depend on identifying a client to which that communication pertains (e.g., based on a content analysis or user input indicative of the client) and determining one or more metrics for each of one or more terminal devices associated with the client.
- each cluster of terminal devices 530a-c can correspond to a different client.
- the terminal devices may be geographically co-located or disperse.
- the metrics may be determined based on stored or learned data and/or real-time monitoring (e.g., based on availability).
- Connection management system 520 can communicate with various terminal devices via one or more routers 525 or other inter-network or intra-network connection components.
- Connection management system 520 may collect, analyze and/or store data from or pertaining to communications, terminal-device operations, client rules, and/or user-associated actions (e.g., online activity) at one or more data stores. Such data may influence communication routing.
- connection management system 520 also is connected to a web server 540.
- connection management system 520 can retrieve data of interest, such as technical item details, and so on.
- Network device 505 may also be connected to a web server (e.g., including a web server 545). In some instances, communication with such a server provided an initial option to initiate a communication exchange with connection management system 520. For example, network device 505 may detect that, while visiting a particular webpage, a communication opportunity is available and such an option can be presented.
- a web server e.g., including a web server 545.
- One or more elements of communication system 500 can also be connected to a socialnetworking server 550.
- Social networking server 550 can aggregate data received from a variety of user devices.
- connection management system 520 may be able to estimate a general (or user-specific) behavior of a given user or class of users.
- FIG. 6 shows a block diagram of an embodiment of a connection management system 600.
- a message receiver interface 605 can receive a message.
- the message can be received, for example, as part of a communication transmitted by a source device (e.g., housed separately from connection management system 600 or within a same housing), such as a network device or terminal device.
- the communication can be part of a series of communications or a communicate exchange, which can include a series of messages or message exchange being routed between two devices (e.g., a network device and terminal device). This message or communication exchange may be part of and/or may define an interaction between the devices.
- a communication channel or operative channel can include one or more protocols (e.g., routing protocols, task-assigning protocols and/or addressing protocols) used to facilitate routing and a communication exchange between the devices.
- the message can include a message generated based on inputs received at a local or remote user interface.
- the message can include a message that was generated based on button or key presses or recorded speech signals.
- the message includes an automatically generated message, such as one generated upon detecting that a network device is presenting a particular app page or webpage or has provided a particular input command (e.g., key sequence).
- the message can include an instruction or request, such as one to initiate a communication exchange.
- the message can include or be associated with an identifier of a client.
- the message can explicitly identify the client (or a device associated with the client); the message can include or be associated with a webpage or app page associated with the client; the message can include or be associated with a destination address associated with a client; or the message can include or be associated with an identification of an item (e.g., product) or service associated with the client.
- a network device may be presenting an app page of a particular client, which may offer an option to transmit a communication to an agent.
- a communication may be generated to include the message and an identifier of the particular client.
- a processing engine 610 may process a received communication and/or message. Processing can include, for example, extracting one or more particular data elements (e.g., a message, a client identifier, a network-device identifier, an account identifier, and so on). Processing can include transforming a formatting or communication type (e.g., to be compatible with a particular device type, operating system, communication-channel type, protocol and/or network).
- a formatting or communication type e.g., to be compatible with a particular device type, operating system, communication-channel type, protocol and/or network.
- a message assessment engine 615 may assess the (e.g., extracted or received) message.
- the assessment can include identifying, for example, one or more categories or tags for the message.
- category or tag types can include (for example) topic, sentiment, complexity, and urgency.
- a difference between categorizing and tagging a message can be that categories can be limited (e.g., according to a predefined set of category options), while tags can be open.
- a topic can include, for example, a technical issue, a use question, or a request.
- a category or tag can be determined, for example, based on a semantic analysis of a message (e.g., by identifying keywords, sentence structures, repeated words, punctuation characters and/or nonarticle words); user input (e.g., having selected one or more categories); and/or message-associated statistics (e.g., typing speed and/or response latency).
- a semantic analysis of a message e.g., by identifying keywords, sentence structures, repeated words, punctuation characters and/or nonarticle words
- user input e.g., having selected one or more categories
- message-associated statistics e.g., typing speed and/or response latency
- message assessment engine 615 can determine a metric for a message.
- a metric can include, for example, a number of characters, words, capital letters, allcapital words or instances of particular characters or punctuation marks (e.g., exclamation points, question marks and/or periods).
- a metric can include a ratio, such as a fraction of sentences that end with an exclamation point (or question mark), a fraction of words that are all capitalized, and so on.
- Message assessment engine 615 can store a message, message metric and/or message statistic in a message data store 620. Each message can also be stored in association with other data (e.g., metadata), such as data identifying a corresponding source device, destination device, network device, terminal device, client, one or more categories, one or more stages and/or message-associated statistics).
- Various components of connection management system 600 e.g., message assessment engine 615 and/or an interaction management engine 625) can query message data store 620 to retrieve query-responsive messages, message metrics and/or message statistics.
- An interaction management engine 625 can determine to which device a communication is to be routed and how the receiving and transmitting devices are to communicate. Each of these determinations can depend, for example, on whether a particular network device (or any network device associated with a particular user) has previously communicated with a terminal device in a set of terminal devices (e.g., any terminal device associated with connection management system 600 or any terminal device associated with one or more particular clients).
- communication routing can be generally biased towards the same terminal device.
- Other factors that may influence routing can include, for example, whether the terminal device (or corresponding agent) is available and/or a predicted response latency of the terminal device. Such factors may be considered absolutely or relative to similar metrics corresponding to other terminal devices.
- a re-routing rule e.g., a client-specific or general rule
- a terminal-device selection can be performed based on factors such as, for example, an extent to which various agents’ knowledge base corresponds to a communication topic, availability of various agents at a given time and/or over a channel type, types and/or capabilities of terminal devices (e.g., associated with the client).
- a rule can identify how to determine a sub-parameter to one or more factors such as these and a weight to assign to each parameter. By combining (e.g., summing) weighted sub-parameters, a parameter for each agent can be determined.
- a terminal device selection can then be made by comparing terminal devices’ parameters.
- interaction management engine 625 can (for example) determine whether a terminal device is to respond to a communication via (for example) SMS message, voice call, video communication, etc.
- a communication type can be selected based on, for example, a communication-type priority list (e.g., at least partly defined by a client or user); a type of a communication previously received from the network device (e.g., so as to promote consistency), a complexity of a received message, capabilities of the network device, and/or an availability of one or more terminal devices.
- some communication types will result in real-time communication (e.g., where fast message response is expected), while others can result in asynchronous communication (e.g., where delays (e.g., of several minutes or hours) between messages are acceptable).
- interaction management engine 625 can determine whether a continuous channel between two devices should be established, used or terminated.
- a continuous channel can be structured so as to facilitate routing of future communications from a network device to a specified terminal device. This bias can persist even across message series.
- a representation of a continuous channel e.g., identifying an agent
- a parameter can be generated using one or more factors described herein and a rule (e.g., that includes a weight for each of the one or more factors) to determine a connection parameter corresponding to a given network device and terminal device.
- the parameter may pertain to an overall match or one specific to a given communication or communication series.
- the parameter may reflect a degree to which a given terminal device is predicted to be suited to respond to a network-device communication.
- a parameter analysis can be used to identify each of a terminal device to route a given communication to and whether to establish, use or terminate a connection channel.
- a parameter relevant to each decision may be determined in a same, similar or different manner.
- different factors may be considered depending on whether the parameter is to predict a strength of a long-term match versus one to respond to a particular message query. For example, in the former instance, considerations of overall schedules and time zones may be important, while in the latter instance, immediate availability may be more highly weighted.
- a parameter can be determined for a single network- device/terminal-device combination, or multiple parameters can be determined, each characterizing a match between a given network device and a different terminal device.
- a set of three terminal devices associated with a client may be evaluated for potential communication routing.
- a parameter may be generated for each that relates to a match for the particular communication.
- Each of the first two terminal devices may have previously communicated with a network device having transmitted the communication.
- An input from the network device may have indicated positive feedback associated with an interaction with the communication(s) with the first device.
- a past-interact sub-parameter (as calculated according to a rule) for the first, second and third devices may be 10, 5, and 0, respectively.
- ft may be determined that only the third terminal device is available, ft may be predicted that the second terminal device will be available for responding within 15 minutes, but that the first terminal device will not be available for responding until the next day.
- a fast-response sub-parameter for the first, second and third devices may be 1, 3 and 10.
- it may be estimated a degree to which an agent (associated with the terminal device) is knowledgeable about a topic in the communication, ft may be determined that an agent associated with the third terminal device is more knowledgeable than those associated with the other two devices, resulting in sub-parameters of 3, 4 and 9.
- the rule does not include weighting or normalization parameters (though, in other instances, a rule may), resulting in parameters of 14, 11 and 19.
- the rule may indicate that the message is to be routed to a device with the highest parameter, that being the third terminal device. If routing to a particular terminal device is unsuccessful, the message can be routed to a device with the next-highest parameter, and so on.
- a parameter may be compared to one or more absolute or relative thresholds. For example, parameters for a set of terminal devices can be compared to each other to identify a high parameter to select a terminal device to which a communication can be routed. As another example, a parameter (e.g., a high parameter) can be compared to one or more absolute thresholds to determine whether to establish a continuous channel with a terminal device.
- An overall threshold for establishing a continuous channel may (but need not) be higher than a threshold for consistently routing communications in a given series of messages. This difference between the overall threshold and threshold for determining whether to consistently route communication may be because a strong match is important in the continuous-channel context given the extended utility of the channel.
- an overall threshold for using a continuous channel may (but need not) be lower than a threshold for establishing a continuous channel and/or for consistently routing communications in a given series of messages.
- Interaction management engine 625 can interact with an account engine 630 in various contexts.
- account engine 630 may look up an identifier of a network device or terminal device in an account data store 635 to identify an account corresponding to the device.
- account engine 630 can maintain data about previous communication exchanges (e.g., times, involved other device(s), channel type, resolution stage, topic(s) and/or associated client identifier), connection channels (e.g., indicating - for each of one or more clients - whether any channels exist, a terminal device associated with each channel, an establishment time, a usage frequency, a date of last use, any channel constraints and/or supported types of communication), user or agent preferences or constraints (e.g., related to terminal-device selection, response latency, terminal-device consistency, agent expertise, and/or communication-type preference or constraint), and/or user or agent characteristics (e.g., age, language(s) spoken or preferred, geographical location, interests, and so on).
- previous communication exchanges e.g.,
- interaction management engine 625 can alert account engine 630 of various connection-channel actions, such that account data store 635 can be updated to reflect the current channel data. For example, upon establishing a channel, interaction management engine 625 can notify account engine 630 of the establishment and identify one or more of: a network device, a terminal device, an account and a client. Account engine 635 can (in some instances) subsequently notify a user of the channel’s existence such that the user can be aware of the agent consistency being availed.
- Interaction management engine 625 can further interact with a client mapping engine 640, which can map a communication to one or more clients (and/or associated brands).
- a communication received from a network device itself includes an identifier corresponding to a client (e.g., an identifier of a client, webpage, or app page).
- the identifier can be included as part of a message (e.g., which client mapping engine 640 may detect) or included as other data in a message-inclusive communication.
- Client mapping engine 640 may then look up the identifier in a client data store 645 to retrieve additional data about the client and/or an identifier of the client.
- a message may not particularly correspond to any client.
- a message may include a general query.
- Client mapping engine 640 may, for example, perform a semantic analysis on the message, identify one or more keywords and identify one or more clients associated with the keyword(s).
- a single client is identified.
- multiple clients are identified.
- An identification of each client may then be presented via a network device such that a user can select a client to communicate with (e.g., via an associated terminal device).
- Client data store 645 can include identifications of one or more terminal devices (and/or agents) associated with the client.
- a terminal routing engine 650 can retrieve or collect data pertaining to each of one, more or all such terminal devices (and/or agents) so as to influence routing determinations.
- terminal routing engine 650 may maintain a terminal data store 655, which can store information such as terminal devices’ device types, operating system, communication-type capabilities, installed applications accessories, geographic location and/or identifiers (e.g., IP addresses). Some information can be dynamically updated.
- information indicating whether a terminal device is available may be dynamically updated based on (for example) a communication from a terminal device (e.g., identifying whether the device is asleep, being turned off/on, non-active/active, or identifying whether input has been received within a time period); a communication routing (e.g., indicative of whether a terminal device is involved in or being assigned to be part of a communication exchange); or a communication from a network device or terminal device indicating that a communication exchange has ended or begun.
- a communication from a terminal device e.g., identifying whether the device is asleep, being turned off/on, non-active/active, or identifying whether input has been received within a time period
- a communication routing e.g., indicative of whether a terminal device is involved in or being assigned to be part of a communication exchange
- a communication from a network device or terminal device indicating that a communication exchange has ended or begun.
- being engaged in one or more communication exchanges does not necessarily indicate that a terminal device is not available to engage in another communication exchange.
- Various factors such as communication types (e.g., message), client-identified or user-identified target response times, and/or system loads (e.g., generally or with respect to a user) may influence how many exchanges a terminal device may be involved in.
- interaction management engine 625 When interaction management engine 625 has identified a terminal device to involve in a communication exchange or connection channel, it can notify terminal routing engine 650, which may retrieve any pertinent data about the terminal device from terminal data store 655, such as a destination (e.g., IP) address, device type, protocol, etc. Processing engine 610 can then (in some instances) modify the message-inclusive communication or generate a new communication (including the message) so as to have a particular format, comply with a particular protocol, and so on. In some instances, a new or modified message may include additional data, such as account data corresponding to a network device, a message chronicle, and/or client data.
- destination e.g., IP
- Processing engine 610 can then (in some instances) modify the message-inclusive communication or generate a new communication (including the message) so as to have a particular format, comply with a particular protocol, and so on.
- a new or modified message may include additional data, such as account data corresponding to a network device, a message chronic
- a message transmitter interface 660 can then transmit the communication to the terminal device.
- the transmission may include, for example, a wired or wireless transmission to a device housed in a separate housing.
- the terminal device can include a terminal device in a same or different network (e.g., local-area network) as connection management system 600. Accordingly, transmitting the communication to the terminal device can include transmitting the communication to an inter- or intra-network connection component.
- bots can be configured to autonomously communicate with network devices. Further, bots can be configured for a specific capability. Examples of capabilities can include updating database records, providing updates to users, providing additional data about the user to agents, determining a user’s intent and routing the user to a destination system based on the intent, predicting or suggesting responses to agents communicating with users, escalating communication sessions to include one or more additional bots or agents, and other suitable capabilities.
- a communication server can automatically and dynamically determine to switch the bot with a terminal device. For example, bots can communicate with users about certain tasks (e.g., updating a database record associated with a user), whereas, terminal devices can communicate with users about more difficult tasks (e.g., communicating using a communication channel to solve a technical issue).
- a network device e.g., operated by the user
- a communication server can automatically and dynamically determine to switch the bot with a terminal device. For example, bots can communicate with users about certain tasks (e.g., updating a database record associated with a user), whereas, terminal devices can communicate with users about more difficult tasks (e.g., communicating using a communication channel to solve a technical issue).
- determining whether to switch between a bot and a terminal device during a communication session can be based on an analysis of one or more characteristics of the messages in a communication session.
- a dynamic sentiment parameter can be generated to represent a sentiment of messages, conversations, entities, agents, and so on. For example, in cases where the dynamic sentiment parameter indicates that the user is frustrated with the bot, the system can automatically switch the bot with a terminal device so that a live agent can communicate with the user. See U.S. Serial No. 15/171,525, filed June 2, 2016, the disclosure of which is incorporated by reference herein in its entirety for all purposes.
- determining whether to switch between the bots and terminal devices can be performed without a prompt from a user.
- the determination can be performed automatically at the communication server based any number of factors, including characteristics of the current messages in the communication session (e.g., chat), characteristics of previous messages transmitted by the user in previous communication sessions, a trajectory of a characteristic (e.g., a sentiment) over multiple messages in a conversation, or additional information associated with the user (e.g., profile information, preference information, and other suitable information associated with the user).
- characteristics of the current messages in the communication session e.g., chat
- characteristics of previous messages transmitted by the user in previous communication sessions e.g., a trajectory of a characteristic (e.g., a sentiment) over multiple messages in a conversation
- additional information associated with the user e.g., profile information, preference information, and other suitable information associated with the user.
- FIG. 7 shows a block diagram of a network environment for dynamically switching between bots and terminal devices during communication sessions.
- network environment 700 can include network device 705, communication server 710, terminal device 715, and bot 720.
- Communication server 710 can be a server with one or more processors with at least one storage device, and can be configured to perform methods and techniques described herein.
- communication server 710 can manage communication sessions between network devices (e.g., operated by users) and terminal devices (e.g., operated by agents).
- Communication server 710 can establish a communication channel between network device 705 and terminal device 715 so that network device 705 and terminal device 715 can communicate with each other during a communication session.
- a communication session can facilitate the exchange of one or more messages between network device 705 and terminal device 715.
- the present disclosure is not limited to the exchange of messages during a communication session.
- Other forms of communication can be facilitated by the communication session, for example, video communication (e.g., a video feed) and audio communication (e.g., a Voice-Over-IP connection).
- communication server 710 can establish a communication channel between network device 705 and bot 720.
- Bot 720 can be code that, when executed, is configured to autonomously communicate with network device 705.
- bot 720 can be a bot that automatically generates messages to initiate conversations with the user associated with network device 705 and/or to automatically respond to messages from network device 705.
- communication server 710 can be associated with a platform.
- Clients e.g., an external system to the platform
- clients can use their own bots in the platform, which enables clients to implement the methods and techniques described herein into their internal communication systems.
- bots can be defined by one or more sources.
- data store 730 can store code representing bots that are defined (e.g., created or coded) by clients of the communication server.
- a client that has defined its own bots can load the bots to the communication server 710.
- the bots defined by clients can be stored in client bots data store 730.
- Data store 740 can store code representing bots that are defined by third-party systems.
- a third-party system can include an independent software vendor.
- Data store 750 can store code representing bots that are defined by an entity associated with communication server 710.
- bots that are coded by the entity can be loaded to or accessible by communication server 710, so that the bots can be executed and autonomously communicate with users.
- communication server 710 can access bots stored in data store 730, data store 740, and/or data store 750 using cloud network 760.
- Cloud network 760 may be any network, and can include an open network, such as the Internet, personal area network, local area network (LAN), campus area network (CAN), metropolitan area network (MAN), wide area network (WAN), wireless local area network (WLAN), a private network, such as an intranet, extranet, or other backbone.
- LAN local area network
- CAN campus area network
- MAN metropolitan area network
- WAN wide area network
- WLAN wireless local area network
- a private network such as an intranet, extranet, or other backbone.
- terminal device 715 can be operated by an agent.
- Terminal device 715 can be any portable (e.g., mobile phone, tablet, laptop) or non-portable device (e.g., electronic kiosk, desktop computer, etc.).
- the agent can access a website using a browser that is running on terminal device 715.
- the website can include a console or platform that is running on the browser of terminal device 715.
- the agent can be logged into the platform using the browser.
- One or more login credentials e.g., username, password, and the like
- One or more login credentials can be used to authenticate the agent’s identity before allowing the agent to gain access to the console or web applications included in the console.
- Examples of a console can include a platform that includes one or more APIs (application programming interfaces), a dashboard including one or more functions, a web-hosted application running on a web browser (without the need for downloading plug-ins) that is capable of establishing or joining a communication session, and other suitable interfaces.
- the console can include one or more web applications or functions that can be executed.
- the web applications or functions can be executed at the browser, at communication server 710, a local server, a remote server, or other suitable computing device.
- the web applications, native applications, or functions can enable an agent to communicate with a user, and to view communications between the user and one or more bots.
- communication server 710 can be configured to dynamically switch between bot 720 and terminal device 715 during a particular communication session.
- communication server 710 can facilitate a communication session between network device 705 and bot 720.
- Bot 720 can be configured to autonomously communicate with network device 705 by exchanging one or more messages with the network device 705 during the communication session.
- Communication server 710 can dynamically determine whether to switch bot 720 with terminal device 715 (or in some cases, vice versa) so that a live agent can communicate with network device 705, instead of bot 720.
- the switching can be performed without a prompt from the network device 705 or terminal device 715.
- the switching can be based on message parameters (e.g., scores representing sentiment of a message or series of messages) of the messages exchanged between the network device 705 and the bot 720, without prompting the network device 705 to request a terminal device.
- communication server 710 can determine to switch between bot 720 and terminal device 715 automatically based on characteristics of the messages exchanged between the bot 720 and the network device 705.
- analyzing the text of a message to determine the characteristic can include an analysis of textual or non-textual attributes associated with the message.
- communication server 710 can extract one or more lines of text included in the message from network device 705.
- Communication server 710 can identify whether the one or more lines of text include an anchor.
- an anchor examples include a string of text associated with a polarity (e.g., sentiment or intent, the word “frustrated” corresponding to a negative polarity or frustrated polarity, the word “happy” corresponding to a positive polarity, and so on).
- a term “dispute” for one client can be negative, but can be neutral or positive for a second client.
- anchors can be dynamically determined using supervised machine learning techniques. For example, one or more clustering algorithms can be executed on stored messages to find patterns within the stored messages. The clustered messages can be further filtered and evaluated to determine the anchor. Further, one or more words near the identified anchor can be parsed for amplifiers.
- An example of an amplifier is a term that increases or decreases an intensity associated with the polarity of the anchor, such as “really,” “not really,” “kind of,” and so on.
- the characteristic can include, for example, the speed of typing, the number of special characters used in the message (e.g., exclamation points, question marks, and so on), a semantic analysis of a message (e.g., by identifying keywords, sentence structures, repeated words, punctuation characters and/or nonarticle words); user input (e.g., having selected one or more categories); and/or message-associated statistics (e.g., response latency).
- the message parameter can be a numerical value that indicates the high intensity of the negative polarity (e.g., a message parameter of 20 on a scale of 0-100, with lower numbers indicating a negative polarity and higher numbers indicating a positive polarity).
- An algorithm can be used to calculate the message parameter.
- the algorithm may be based on supervised machine learning techniques.
- the term “kind of’ is near the anchor “don’t like” (e.g., as in the sentence “I kind of don’t like”), the term “kind of’ may be identified as an amplifier term that indicates a medium intensity of the negative polarity.
- a message parameter can be generated based on the identification of the medium intensity of the negative polarity.
- the message parameter can be a numerical value that indicates the medium intensity of the negative polarity (e.g., a message parameter of 40, as opposed to the message parameter of 20).
- the message parameter can be used to determine which secondary queue is to store the communication.
- the characteristic of a message can be the sentiment associated with the message.
- communication server 710 may recommend or predict responses to messages received from network device 705.
- communication server 710 can include a message recommendation system, which can evaluate messages received from network device 705 and use a machine-learning model to recommend responses to those received messages.
- the message recommendation system can display a set of recommended messages on terminal device 715 to assist the agent in communicating with network device 705.
- FIG. 8 shows a block diagram representing network environment 800 for dynamically selecting endpoints across multiple communication channels.
- network environment 800 may include network device 805, terminal device 810, and communication server 820.
- Network device 805 may be similar to network device 705, and thus, a description is omitted here for the sake of brevity.
- Terminal device 810 may be similar to terminal device 715, and thus, a description is omitted here for the sake of brevity.
- Communication server 820 may be similar to Communication server 710, and thus, a description is omitted here for the sake of brevity.
- Communication server 820 may establish or facilitate the establishment of a communication channel between network device 805 and terminal device 810. As illustrated in FIG. 8, communication server 820 may establish communication channel C 840, which enables network device 805 and terminal device 810 to exchange one or more messages. As a non-limiting example, communication channel C 840 may be a web chat feature of a website, communication channel B 835 may be a chat application running on a mobile device (e.g., a smart phone), and communication channel A 830 may be a voice over Internet Protocol (VOIP) audio channel that enables the agent to communicate with the user. [00106] Communication server 820 may configure bot 825 to autonomously communicate with network device 805.
- VOIP voice over Internet Protocol
- bot 825 may access and execute one or more protocols that enable bot 825 to communicate with network device 805 using communication channel C 840.
- bot 825 may access and execute a protocol for communicating over the web chat feature of the website.
- the protocol may include a coding language specific to the web chat feature for exchanging messages using the web chat feature.
- the protocol may include code that, when executed, converts a message (e.g., a string of text or other content) inputted by an agent at terminal device 810 into structured content (e.g., content separated into independent data fields), and maps the structured content to elements of the web chat feature of the website.
- bot 825 can translate the structured content to the elements of the web chat feature to enable the message to be communicated using the web chat feature.
- bot 825 can also be configured to communicate with network device 805 using communication channel B 835.
- Communication channel B 835 can be a different communication channel from communication channel C 840. Further, communication channel B 835 may require different elements to facilitate communication than the elements required for communication channel C 840.
- Bot 825 can be configured to translate the structured content to the elements of communication channel B 835.
- communication channel B 835 may be an in-app chat feature of a native application running on a smart phone.
- One or more elements may be required in order to facilitate communication using communication channel B 835.
- FACEBOOK MESSENGER may be the native application running on the smart phone.
- the one or more elements of FACEBOOK MESSENGER may be templates specific to FACEBOOK MESSENGER that are required to facilitate communication using FACEBOOK MESSENGER.
- the protocol that enables bot 825 to communicate using communication channel B 835 may map the structured content to the templates of the FACEBOOK MESSENGER native application in order to transmit the structured content as a message within the FACEBOOK MESSENGER application.
- a mobile application may include executable code (stored in the mobile device or at one or more external servers) that can be executed using the operating system of the network device (e.g., a smartphone).
- the mobile application may include a hybrid mobile application that is comprised of native user interface (UI) components (generated and stored at the mobile device), but is written in an interpreted language (e.g., using Web-based coding languages).
- UI native user interface
- the present disclosure is not limited to mobile native applications or hybrid applications, and thus, any type of mobile application may be used in the methods described herein.
- bot 825 can also be configured to communicate with network device 805 using communication channel A 830.
- Communication channel A 835 can be a different communication channel from communication channel C 840 and communication channel B 835.
- communication channel A 830 may require different elements to facilitate communication than the elements required for communication channel C 840 and for communication channel B 835.
- Bot 825 can be configured to translate the structured content to the elements of communication channel A 830.
- communication channel A 830 may be a VOIP audio communication link between network device 805 and terminal device 810.
- One or more elements may be required in order to facilitate communication using communication channel A 830.
- the protocol may include a mapping of the structured content to the elements associated with communication channel A 830.
- communication server 820 may be configured to dynamically, autonomous, and/or automatically transfer a communication session between different communication channels, so that bot 825 can continuously communicate with network device 805, regardless of the communication channel.
- network device 805 may be communicating with terminal device 810 using a first communication channel 845 (i.e., communication channel C 840).
- Network device 805 may transmit a message indicating that the user operating network device 805 intends to change the communication channel currently being used for the communication session.
- network device 805 may indicate that second communication channel 850 is the target communication channel for continuing the communication session with terminal device 810.
- Bot 825 can automatically detect the indication that the communication channel should be changed from first communication channel 845 to second communication channel 850.
- bot 825 may continuously evaluate messages exchanged during the communication session to detect that the communication channel should be changed.
- communication server may identify the user identifier associated with network device 805.
- user data database 815 may store user identifiers for various users.
- a user identifier may be a string of text and/or numbers that uniquely identifies a network device. If, at any given time, communication server 820 determines that the same user identifier is associated with two active communication channels, communication server 820 can recognize that the network device is requesting to continue a communication session but to change the communication channels.
- Communication server 820 may be configured to support continuity between different communication channels.
- the target communication channel e.g., second communication channel 850
- bot 825 can be automatically used by bot 825 to continue the communication session with network device 805, but using second communication channel 850, instead of first communication channel 845.
- bot 825 may automatically transmit a message to network device 805 using second communication channel 850. Transmitting the message to network device 805 may indicate to network device 805 that the transfer of communication channels is complete.
- communication server 820 may automatically detect that the communication channel has been changed from first communication channel 845 to second communication channel 850.
- communication server 820 may recognize the user identifier associated with network device 805 when network device 805 is communicating with bot 825 using first communication channel 845. If network device 805 begins using second communication channel 850 (e.g., without indicating the intention to change communication channels during the communication session), communication server 820 can automatically detect that the user identifier for network device 805 is currently associated with two active communication channels (e.g., first communication channel 845 and second communication channel 850). Communication server 820 can detect that first communication channel 845 is associated with a recent history of messages (e.g., messages transmitted or exchanged within the last five minutes) and that second communication channel 850 is not associated with a recent history of messages (e.g., within the last few minutes).
- first communication channel 845 is associated with a recent history of messages (e.g., messages transmitted or exchanged within the last five minutes) and that second communication channel 850 is not associated with a recent history of messages (e.g., within the last few minutes).
- communication server 820 can determine that network device 805 is requesting to transfer the communication session from first communication channel 845 to second communication channel 850.
- Communication server 820 can implement the transfer by accessing the protocol associated with second communication channel 850, and executing bot 825 using the accessed protocol to enable bot 825 or terminal device 810 to communicate with network device 805 using second communication channel 850, instead of using first communication channel 845.
- one or more machine-learning techniques can be used to identify patterns in the communication channel usage of network device 805.
- the usage of communication channels by network device 805 can be tracked and recorded (and stored as historical data).
- Machine-learning techniques can be applied to the historical data to identify which communication channel network device 805 is most likely to use when communicating with a particular entity (e.g., bot, company, terminal device, agent, and so on).
- a particular entity e.g., bot, company, terminal device, agent, and so on.
- communication server 820 can establish a communication channel of the type that network device 805 is most likely to use (based on the results of the machine learning techniques). As network device 805 begins to use a different communication channel more frequently, communication server 820 can identify this changing trend and initiate communication sessions using the most used or most frequently used communication channel.
- FIG. 9 shows a block diagram representing network environment 900 for enhancing endpoint selection using machine-learning techniques.
- Network environment 900 may include network device 905 (operated by a user) communication server 910, bot 915 and terminal device 920.
- Communication server 910 can facilitate the establishment of a communication channel that enables network device 905 and at least one bot 915 and terminal device 920 to communication.
- Communication server 910 may include intelligent routing system 925, message recommendation system 930, and message data store 935.
- Each of intelligent routing system 925 and message recommendation system 930 may include one or more computing devices with a processor and a memory that execute instructions to implement certain operations.
- intelligent routing system 925 may be a bot configured to intelligently route communications received from network devices to the appropriate destination.
- Intelligent routing system 925 may include one or more processors configured to execute code that causes one or more machine-learning techniques or artificial intelligence techniques to intelligently route messages.
- intelligent routing system 925 can execute one or more machine-learning techniques to train a model that predicts a destination associated with a message received from network device 905.
- intelligent routing system 925 may receive a message from network device 905 through a communication channel established or facilitated by communication server 910 (e.g., a native application configured to enable users to communicate with each other across various devices).
- Intelligent routing system 925 may evaluate the incoming message according to certain embodiments described above.
- intelligent routing system 925 may evaluate the content (e.g., text, audio clips, images, emoticons, or other suitable content) included in the received message using a trained machine-learning model.
- the content of the message can be inputted into the machine-learning model to generate a predicted destination (e.g., a particular terminal device or bot).
- the machine-learning model may be continuously trained based on feedback signal 940 received from network device 905.
- intelligent routing system 925 may request an acknowledgement from network device 905 of the predicted destination.
- intelligent routing system 925 may evaluate the message using a machine-learning technique, and a result of the evaluation may include a predication that bot 915 is the destination for the message. To confirm, intelligent routing system 925 may automatically request feedback signal 940.
- feedback signal 940 may include a request for network device 905 to acknowledge whether bot 915 is the correct destination for the message (e.g., “Is Technical Support the correct destination?”).
- intelligent routing system 925 may train the machinelearning model to predict that future messages including the exact or similar content (e.g., a threshold of similarity, such as 10 percent difference in content) as the received message are to be routed to bot 915. However, if intelligent routing system 925 receives feedback signal 940 indicating that bot 915 is not the correct or intended destination for the received message, but rather terminal device 920 was the correct or intended destination, intelligent routing system 925 can train the machine-learning model that future messages including the exact or similar content as the received message are to be routed to terminal device 920 (instead of bot 915).
- a threshold of similarity such as 10 percent difference in content
- intelligent routing system 925 may not immediately update or train the machinelearning model to route future messages to terminal device 920, but rather, intelligent routing system 925 may wait for a threshold number of incorrect routings to bot 915 before routing all future messages with the exact same or similar content as the received message to terminal device 920. As a non-limiting example, intelligent routing system 925 may begin routing future messages (that were predicted to be routed to bot 915) to terminal device 920 instead of bot 915 after five instances of network devices transmitting feedback signals indicating that bot 915 is not the correct or intended destination.
- intelligent routing system 925 may select where to route a given message based on bids received to handle a particular request in the message. Intelligent routing system 925 may broadcast an intent to disparate services and determine who wants to bid on handling the request. Bidding parties may respond with their level of confidence in successfully handling the request and a plan to execute handling of the request. Intelligent routing system 925 may evaluate all of the responses from the bidding parties and, based on machine learning policies, determine which bidding party to use for a given message.
- Message data store 935 may store some (e.g., but not all) or all messages received in the past from one or more network devices. Further, message data store 935 may also store some or all messages transmitted by terminal devices or bots during previous communication sessions with network devices. Message data store 935 may also store some or all messages transmitted by network devices to bots during communication sessions. Further, message data store 935 may store some or all messages transmitted by bots to network devices during communication sessions. In some implementations, message data store 935 may be a database of all messages processed (e.g., transmitted by or received at) communication server 910.
- Message recommendation system 930 may analyze the database of messages stored at message data store 935.
- message recommendation system 930 may evaluate the messages stored at message data store 935 using one or more machine-learning algorithms or artificial intelligence algorithms.
- message recommendation system 930 may execute one or more clustering algorithms, such as K-means clustering, means-shift clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering, Expectation-Maximization (EM) Clustering using Gaussian Mixture Models (GMM), and other suitable machine-learning algorithms, on the database of messages stored in message data store 935.
- clustering algorithms such as K-means clustering, means-shift clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering, Expectation-Maximization (EM) Clustering using Gaussian Mixture Models (GMM), and other suitable machine-learning algorithms, on the database of messages stored in message data store 935.
- a recurrent neural network (RNN) or a convolutional neural network (CNN) may be used to predict response messages to assist the agent.
- message recommendation system 930 may use support vector machines (SVM), supervised, semi-supervised, ensemble techniques, or unsupervised machine-learning techniques to evaluate all previous messages to predict responses to incoming messages received from network devices during communication sessions. For example, message recommendation system 930 may evaluate the content of messages received from network devices (or messages received at communication server 910 from bots or terminal devices) and compare the results of the evaluation to the one or more clusters of previous messages stored in message data store 935. Once the cluster is identified, message recommendation system 930 can identify the most relevant response messages based on a confidence threshold.
- an incoming message (e.g., received at communication server 910 from network device 905) may correspond to a technical issue based on the content of the incoming message.
- Message recommendation system 930 can identify that the incoming message corresponds to a technical issue based on an evaluation of the content of the incoming message (e.g., text evaluation).
- Message recommendation system 930 can access message data store 935 to identify the cluster of messages associated with technical issues.
- Message recommendation system 930 can select one or more responses messages within the cluster of messages based on a confidence threshold.
- a confidence algorithm can be executed to generate a confidence score.
- a confidence score may be a percentage value where the lower the percentage, the less likely the response is a good prediction for the incoming message, and the higher the percentage, the more likely the response is a good prediction for the incoming message.
- a minimum confidence threshold may be defined as a measure of certainty or trustworthiness associated with each discovered pattern.
- an example of a confidence algorithm may be the Apriori Algorithm, similarity algorithms indicating similarity between two data sets, and other suitable confidence algorithms.
- FIG. 10 shows an example process for supervising bot operation and training.
- certain criteria may be defined and stored in memory. Such criteria may be applicable to conversations conducted between a bot (or set of bots) and customers.
- the conversation is escalated to an agent device associated with a human agent.
- the bot may make a prediction regarding a specific intent of the customer (e.g., purchasing or returning a specific product, need to update an address, password, or other user information).
- the agent may review the conversation and provide input that may classify or re-classify the conversation.
- the conversation may be handed back to the bot to continue conducting in view of the classification input.
- the classification input may be used to update a learning model associated with the hot, and such updated learning model may be used to conduct subsequent conversations.
- a set of supervision criteria may be stored in memory regarding the set of supervision criteria including a threshold level of confidence.
- Such stored criteria define factors (and levels thereof) that may be measured and considered in making the decision to escalate a conversation that is currently being conducted with a bot.
- the set of supervision criteria may specify one or more conditions, for example, used to determine the threshold level of confidence.
- the specified conditions may be detected within the conversation, as well as counted, measured, or otherwise evaluated and compared to the threshold of confidence. Different aspects of a messages or sets of messages within a conversation may be evaluated against the stored criteria.
- different sets of criteria may be applied to different bots, departments, geographic locations, and systems. Such criteria may also be customized to specific business entities or industries.
- one or more messages in a conversation between a bot and a user of a client device may be monitored and evaluated against the set of supervision criteria stored in step 1005. Such monitoring may occur for specific bots, specific groups of bots, specific departments, or throughout the system. The designation of what to monitored may be customized to different entities or businesses, as well as customized to the associated set of agents. Messages associated with a new type of bot, for example, may be designated for monitoring.
- the monitored messages may be further passed to predict one or more intents regarding the customer’s specific needs.
- certain conversations cues may indicate an intent associated with product information, purchase, and updating user information.
- intents may drive the bot’s coordination of the conversation and result, for example, in routing the conversations to bots respectively tasked with providing product information, purchase orders, and processing updated user information.
- Each prediction regarding user intent may be scored (e.g., a confidence score) based on the stored criteria, and that score may be compared to the threshold level of confidence.
- one or more of the bot predictions regarding user intent may meet at least a set threshold level of confidence, and the bot may proceed with conducting the conversation in a manner that meets the customer’s needs without requiring intervention.
- a predicted intent for particular conversation may fail to at least meet the threshold level of confidence. For example, a particular intent prediction by the hot may fail to meet the threshold level of confidence when a confidence score assigned to that intent prediction is compared to the threshold level of confidence and found not to meet that threshold level of confidence.
- a request for input may be sent to an agent device, which may respond by providing classification input regarding the conversation.
- the request may specifically request input regarding a specific intent prediction that does not meet the threshold level of confidence (e.g., from among one or more other identified intent predictions that do meet the threshold level of confidence).
- the request may present at least a portion of the conversation associated with the specific intent prediction that failed to meet the threshold level of confidence.
- Some embodiments may further present a set of options associated with the request for input. Such options may be associated with the different types of bots (or other parameter of the conversation needed to refine the intent prediction.
- the input from the agent device may be provided to the bot, which may use such input to classify and route the conversation to one or more endpoint devices (e.g., other bots or agents).
- endpoint devices e.g., other bots or agents.
- Such classification and routing may be the same or differ from the identified prediction that failed to meet the threshold level of confidence.
- the conversation may be handed off to a bot designed to intake purchase orders.
- the routing and handoff of the conversation to the one or more endpoints may occur in real-time as the conversation is on-going. For example, the bot may continue asking follow up questions regarding one or more potential intents even as the conversation has been escalated to the agent device.
- step 1035 it may be determined whether any new annotations may be ready to add to a learning model associated with the bot. If not, the method may return to step 1010 in which more conversations continue to be monitored. If there are annotations ready to add to the learning model, the method may proceed to step 1040.
- the classification input from the agent device may also be saved and tracked in the learning model for use by the same bot, same type of bots, and/or other system bots in future conversations. Such classification input may serve, for example, as annotations to the conversation.
- the learning model may be updated based on a single annotated conversation or based on multiple conversations together that have been annotated in the same or similar ways in accordance with the same or similar classification input.
- the new classification input may be added to the learning model for use in future conversations.
- a bot conducting similar conversations in the future may no longer make predictions that fail to meet the threshold level of confidence. As such, similar conversations in the future are therefore not escalated to the agent device. Instead, the bot may use the learning model (now updated based on the classification input from the agent device) in real-time when conducting the next conversation to reach an intent prediction (e.g. , in accordance with the intent prediction or routing decision of step 1030.
- intent prediction may further be assigned a classification score that at least meets the threshold level of confidence.
- FIG. 11 shows a block diagram representing a network environment in which a system for supervising bot operation and training may be implemented.
- the network environment may be categorized by operations respectively associated with customer devices 1105, bot dialogue management 1110, bot intent recognition 1115, bot supervision 1120, and agent devices 1125.
- a conversation may involve and be routed among a plurality of different devices and systems based on real-time analyses of incoming communications.
- Bot dialogue management operations may include functions involved in communicating with customers via any mode known in the art, including telephonic, textual (e.g. , SMS, MMS, instant message), graphic, etc.
- Such dialogue management operations may include greeting customers , asking questions or providing prompts for specific customer information, and generating responses to the customers.
- Response generation may be based on learning models for dialogue management, as well as based on input (e.g. , from agent devices) relating to dialogue management and classification.
- Data regarding conversations may be tracked on an ongoing basis in real-time and stored in conjunction with historical data. The historical and real-time data may also be incorporated into the learning models, which are thereby dynamically updated in view of the same. Such updates to the learning models may further result in dynamic adjustment to predictions and recommendations made for similar conversations over time.
- Different learning models may also be customized to different entities (e.g., companies, brands) operating the bots, different customers, and categories of the same.
- the intent recognition operations may include text classification, dialogue annotation in relation to intent recognition, and training intent-related models. Meanwhile, the customer may be prompted for such information as may initiate a conversation, interaction with predefined contexts (e.g., buttons, drop-down menus), and natural language communications. As noted above, such information may be provided through a variety of different types of communication channels. Regardless of channel, such customer input may be parsed, analyzed, and used to perform operations related to bot dialogue management 1110, intent recognition 1115, and bot supervision 1120 in real-time.
- predefined contexts e.g., buttons, drop-down menus
- a customer device 1105 may initiate a conversation within a communication system.
- the customer device 1105 may initiate a conversation through any of the channels discussed above. Such initiation may be made expressly (e.g. , request or selection of an option) or implicitly (e.g., repeated searches, extended browsing).
- a communication session may be established between customer device 1105 and a chatbot (e.g., chatbot dialogue management system 1110).
- chatbot dialogue management system 1110 may respond to initiation of the conversation or establishment of the communication session by transmitting a greeting or other prompt to the customer device.
- the communication session may be associated with a graphic user interface with which the customer of the customer device 1105 may make selections (e.g. , button clicks, from dropdown menus) or interact with predefined content in step 1140.
- the chatbot dialogue management system 1110 may respond to such selections or interactions based on predetermined dialogue management rules and policies.
- the customer of the customer device 1105 may enter natural language communications such as in step 1150.
- chatbot intent recognition system 1115 may classify the natural language communications in accordance with one or more learning models.
- Such learning models may include both historical and real-time data regarding multiple different conversations.
- the data may include data regarding the specific customer, any chatbot or agent involved, operating entity, keywords and phrases, actual messages exchanged, routing decisions, conversation outcomes, and other contextual data.
- step 1160 it is determined whether confidence scores associated with the classifications satisfy specified criteria or thresholds. If so, the method may proceed to step 1165 in which the chatbot dialogue management system 1110 may respond to the natural language communication based on the classification. If the confidence scores associated with the classifications do not satisfy specified criteria or thresholds, however, the method may proceed to step 1170 in which it is determined whether the chatbot dialogue management system 1110 currently engaged in the conversation is able to service user text. If chatbot dialogue management system 1110 currently engaged in the conversation cannot service user text, the method may proceed to step 1180 in which the conversation is routed to another bot or agent.
- step 1175 in which a live agent using the agent device 1125 is queried in real-time regarding classification of the natural language communication entered by the customer of customer device 1105.
- a natural language communication cannot be categorized by chatbot intent recognition system 1115 at a certain level of confidence (e.g., 80%, 90%)
- the natural language communication may be provided to agent device 1125 for real-time classification.
- the classification provided by the live agent of agent device 1125 may be provided to chatbot dialogue management system 1110 as the method reverts to step 1165 for continued conversation.
- the classification provided by the live agent of agent device 1125 may also be added to a database for storage in step 1185.
- the classification from the live agent of agent device 1125 may also be used to update one or more learning models in real-time, such that other ongoing conversations — with similar natural language communication — may be classified at a higher level of confidence than before the update.
- the live agent of agent device 1125 may also provide annotations to the natural language communication or other conversational data. Such annotations may provide yet further context and supplemental information for understanding the intent
- step 1190 it may be determined whether there were any new annotations associated with the classification. If no, the method may end. If there are any new annotations associated with the classification, the method may proceed to step 1195 in which the annotations are added to data used for intent recognition and classification of natural language communications. Then in step 1199, any learning models reliant on such data may be retrained in real-time based on the classification and associated annotations
- Bot supervision operations may include the operations described in relation to FIG. 11, including evaluating confidence scores, comparing confidence scores to designated thresholds, and adding new classification input and new annotations to learning model databases.
- Bot supervision may further operate in conjunction with input from human agents, such may include text classification, which may directly or indirectly be used to provide routing instructions for the current conversation.
- input may be used to update learning models and database, so that the same classification inputs may not be required from human agents for future conversations conducted by bots.
- FIG. 12 illustrates an exemplary interface for supervising bot operation and training.
- Such an interface may be presented on the agent device in response to the request from a bot.
- such an interface may include several portions: a list of different conversations 1210 (e.g. , associated with bot escalations), a detailed view of one particular conversation may be shown (along with routing options) in an agent supervision graphic user interface 1220, detailed information regarding a particular campaign (e.g. , business) associated with the conversation 1230, and a summary GUI 1240.
- a list of different conversations 1210 e.g. , associated with bot escalations
- a detailed view of one particular conversation may be shown (along with routing options) in an agent supervision graphic user interface 1220
- detailed information regarding a particular campaign e.g. , business
- the list of different conversations 1210 may have been routed to an agent device 1125 based on a current natural language communication having a classification that did not meet certain criteria or thresholds. Further, as illustrated in the agent supervision GUI 1220 portion of the interface, the natural language communication may be presented with a set of different options (e.g., corresponding to different classifications). The options may be selectable to classify the natural language communication in real-time, as the classification may be provided to the chatbot dialogue management system 1110 to formulate a response based on the classification and thus continue the conversation. Contextual data 1230 may also be provided in the interface, so that the agent of agent device 1125 may refer to the same in making the classification. Such contextual data may include data regarding a particular campaign, data regarding the specific conversation, or personal data regarding the customer.
- a human agent selects one of the options presented within the agent supervision GUI 1220, such option may thereafter be used to classify an intent prediction associated with the conversation.
- Summary GUI 1240 portion of the interface may be used to enter annotations associated with the classification.
- the conversation may further be handed back to the bot to operate in accordance with the selected option for intent prediction.
- selection may be analyzed in the context of similar conversations and used to update an associated learning model used by bot in future conversations.
- the classification and annotations may be used to refine learning models for message classification and intent recognition.
- Such learning models may be applied to ongoing and future conversations, which may thereafter result in different classifications and confidence levels for such classifications.
- the disclosed system and methods for bot supervision can be performed using a computing server.
- An example computing server can include a processor (e.g., a central processing unit), memory, non-volatile memory, and an interface device.
- the memory may store data and/or and one or more code sets, software, scripts, etc.
- the components of the computer server can be coupled together via a bus or through some other known or convenient device.
- the processor may be configured to carry out all or part of methods described herein for example by executing code for example stored in memory.
- One or more of a user device or computer, a provider server or server, or a suspended database update server may include the components of the computing server or variations on such a server.
- the computer server taking any suitable physical form, including, but not limited to a Point-of-Sale server (“POS”).
- the computer server may be an embedded computer server, a server-on-chip (SOC), a single-board computer server (SBC) (such as, for example, a computer-on-module (COM) or server-on-module (SOM)), a desktop computer server, a laptop or notebook computer server, an interactive kiosk, a mainframe, a mesh of computer servers, a mobile telephone, a personal digital assistant (PDA), a server, or a combination of two or more of these.
- SOC server-on-chip
- SBC single-board computer server
- COM computer-on-module
- SOM server-on-module
- desktop computer server a laptop or notebook computer server
- mainframe such as, for example, a computer-on-module (COM) or server-on-module (SOM)
- PDA personal digital assistant
- the computer server may include one or more computer servers; be unitary or distributed; span multiple locations; span multiple machines; and/or reside in a cloud, which may include one or more cloud components in one or more networks.
- one or more computer servers may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein.
- one or more computer servers may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein.
- One or more computer servers may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
- the processor may be, for example, be a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor.
- Intel Pentium microprocessor or Motorola power PC microprocessor.
- machine-readable (storage) medium or “computer-readable (storage) medium” include any type of device that is accessible by the processor.
- the memory can be coupled to the processor by, for example, a bus.
- the memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM).
- RAM random access memory
- DRAM dynamic RAM
- SRAM static RAM
- the memory can be local, remote, or distributed.
- the bus can also couples the processor to the non-volatile memory and drive unit.
- the non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer.
- the non-volatile storage can be local, remote, or distributed.
- the non-volatile memory is optional because servers can be created with all applicable data available in memory.
- a typical computer server will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.
- Software can be stored in the non-volatile memory and/or the drive unit. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory herein. Even when software is moved to the memory for execution, the processor can make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution.
- a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers), when the software program is referred to as “implemented in a computer-readable medium.”
- a processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
- the bus can also couples the processor to the network interface device.
- the interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer server.
- the interface can include an analog modem, Integrated Services Digital network (ISDN0 modem, cable modem, token ring interface, satellite transmission interface (e.g., “direct PC”), or other interfaces for coupling a computer server to other computer servers.
- the interface can include one or more input and/or output (I/O) devices.
- the I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device.
- the display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device.
- CTR cathode ray tube
- LCD liquid
- the computer server can be controlled by operating server software that includes a file management server, such as a disk operating server.
- operating server software with associated file management server software is the family of operating servers known as Windows® from Microsoft Corporation of Redmond, WA, and their associated file management servers.
- Windows® from Microsoft Corporation of Redmond, WA
- WindowsTM WindowsTM operating server
- the file management server can be stored in the non-volatile memory and/or drive unit and can cause the processor to execute the various acts required by the operating server to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
- the server operates as a standalone device or may be connected (e.g., networked) to other servers. In a networked deployment, the server may operate in the capacity of a server or a client server in a client-server network environment, or as a peer server in a peer-to-peer (or distributed) network environment.
- the server may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any server capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that server.
- PC personal computer
- PDA personal digital assistant
- machine-readable medium or machine -readable storage medium is shown, by way of example, to be a single medium, the term “machine -readable medium” and “machine- readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
- the term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the server and that cause the server to perform any one or more of the methodologies or modules of disclosed herein.
- routines executed to implement the implementations of the disclosure may be implemented as part of an operating server or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.”
- the computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.
- machine-readable storage media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.
- recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.
- CD ROMS Compact Disk Read-Only Memory
- DVDs Digital Versatile Disks
- operation of a memory device may comprise a transformation, such as a physical transformation.
- a physical transformation may comprise a physical transformation of an article to a different state or thing.
- a change in state may involve an accumulation and storage of charge or a release of stored charge.
- a change of state may comprise a physical change or transformation in magnetic orientation or a physical change or transformation in molecular structure, such as from crystalline to amorphous or vice versa.
- a storage medium typically may be non-transitory or comprise a non-transitory device.
- a non-transitory storage medium may include a device that is tangible, meaning that the device has a concrete physical form, although the device may change its physical state.
- non-transitory refers to a device remaining tangible despite this change in state.
- connection means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or any combination thereof.
- any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices.
- a software module is implemented with a computer program object comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
- Examples may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer.
- Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer server bus.
- any computing servers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
- Examples may also relate to an object that is produced by a computing process described herein.
- Such an object may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any implementation of a computer program object or other data combination described herein.
- Client devices, network devices, and other devices can be computing servers that include one or more integrated circuits, input devices, output devices, data storage devices, and/or network interfaces, among other things.
- the integrated circuits can include, for example, one or more processors, volatile memory, and/or non-volatile memory, among other things.
- the input devices can include, for example, a keyboard, a mouse, a key pad, a touch interface, a microphone, a camera, and/or other types of input devices.
- the output devices can include, for example, a display screen, a speaker, a haptic feedback server, a printer, and/or other types of output devices.
- a data storage device such as a hard drive or flash memory, can enable the computing device to temporarily or permanently store data.
- a network interface such as a wireless or wired interface, can enable the computing device to communicate with a network
- Examples of computing devices include desktop computers, laptop computers, server computers, hand-held computers, tablets, smart phones, personal digital assistants, digital home assistants, as well as machines and apparatuses in which a computing device has been incorporated.
- computer-readable medium includes, but is not limited to, portable or nonportable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and/or data.
- a computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non- transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices.
- a computer-readable medium may have stored thereon code and/or machineexecutable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
- a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.
- the various examples discussed above may further be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
- the program code or code segments to perform the necessary tasks may be stored in a computer-readable or machine-readable storage medium (e.g., a medium for storing program code or code segments).
- a processor(s), implemented in an integrated circuit, may perform the necessary tasks.
- Such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.
- programmable electronic circuits e.g., microprocessors, or other suitable electronic circuits
- the techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above.
- the computer-readable data storage medium may form part of a computer program product, which may include packaging materials.
- the computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like.
- RAM random access memory
- SDRAM synchronous dynamic random access memory
- ROM read-only memory
- NVRAM non-volatile random access memory
- EEPROM electrically erasable programmable read-only memory
- FLASH memory magnetic or optical data storage media, and the like.
- the techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.
- the program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
- DSPs digital signal processors
- ASICs application specific integrated circuits
- FPGAs field programmable logic arrays
- a general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
- a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for implementing a suspended database update server. [00188] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed.
Abstract
Description
Claims
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
IL301077A IL301077A (en) | 2020-09-15 | 2021-09-15 | Bot supervision |
CA3192505A CA3192505A1 (en) | 2020-09-15 | 2021-09-15 | Bot supervision |
JP2023516792A JP2023543698A (en) | 2020-09-15 | 2021-09-15 | bot monitoring |
CN202180076640.9A CN116508040A (en) | 2020-09-15 | 2021-09-15 | Robot supervision |
EP21790696.5A EP4214658A1 (en) | 2020-09-15 | 2021-09-15 | Bot supervision |
AU2021343437A AU2021343437A1 (en) | 2020-09-15 | 2021-09-15 | Bot supervision |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063078718P | 2020-09-15 | 2020-09-15 | |
US63/078,718 | 2020-09-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022060792A1 true WO2022060792A1 (en) | 2022-03-24 |
Family
ID=78087554
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2021/050401 WO2022060792A1 (en) | 2020-09-15 | 2021-09-15 | Bot supervision |
Country Status (8)
Country | Link |
---|---|
US (2) | US11522818B2 (en) |
EP (1) | EP4214658A1 (en) |
JP (1) | JP2023543698A (en) |
CN (1) | CN116508040A (en) |
AU (1) | AU2021343437A1 (en) |
CA (1) | CA3192505A1 (en) |
IL (1) | IL301077A (en) |
WO (1) | WO2022060792A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP4214658A1 (en) | 2020-09-15 | 2023-07-26 | Liveperson, Inc. | Bot supervision |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180322403A1 (en) * | 2017-05-05 | 2018-11-08 | Liveperson, Inc. | Dynamic response prediction for improved bot task processing |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP4214658A1 (en) | 2020-09-15 | 2023-07-26 | Liveperson, Inc. | Bot supervision |
-
2021
- 2021-09-15 EP EP21790696.5A patent/EP4214658A1/en active Pending
- 2021-09-15 US US17/475,725 patent/US11522818B2/en active Active
- 2021-09-15 JP JP2023516792A patent/JP2023543698A/en active Pending
- 2021-09-15 WO PCT/US2021/050401 patent/WO2022060792A1/en active Application Filing
- 2021-09-15 CN CN202180076640.9A patent/CN116508040A/en active Pending
- 2021-09-15 IL IL301077A patent/IL301077A/en unknown
- 2021-09-15 CA CA3192505A patent/CA3192505A1/en active Pending
- 2021-09-15 AU AU2021343437A patent/AU2021343437A1/en active Pending
-
2022
- 2022-10-18 US US17/968,252 patent/US11888792B2/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180322403A1 (en) * | 2017-05-05 | 2018-11-08 | Liveperson, Inc. | Dynamic response prediction for improved bot task processing |
Non-Patent Citations (1)
Title |
---|
MOHAMMAD FORKAN ABDUR RAHIM ET AL: "ECHO: A Tool for Empirical Evaluation Cloud Chatbots", 2020 20TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND INTERNET COMPUTING (CCGRID), 1 May 2020 (2020-05-01), pages 669 - 672, XP055869269, ISBN: 978-1-7281-6095-5, Retrieved from the Internet <URL:https://ieeexplore.ieee.org/stampPDF/getPDF.jsp?tp=&arnumber=9139701> DOI: 10.1109/CCGrid49817.2020.00-26 * |
Also Published As
Publication number | Publication date |
---|---|
CA3192505A1 (en) | 2022-03-24 |
IL301077A (en) | 2023-05-01 |
AU2021343437A1 (en) | 2023-04-13 |
US11888792B2 (en) | 2024-01-30 |
EP4214658A1 (en) | 2023-07-26 |
CN116508040A (en) | 2023-07-28 |
JP2023543698A (en) | 2023-10-18 |
US20230143863A1 (en) | 2023-05-11 |
US11522818B2 (en) | 2022-12-06 |
US20220086108A1 (en) | 2022-03-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2020241751B2 (en) | Dynamic communications routing to disparate endpoints | |
CN114223185A (en) | System and method for transferring messaging to automation | |
US11716261B2 (en) | Systems and methods for external system integration | |
US20240022536A1 (en) | Dynamic communications routing to disparate endpoints | |
US11888792B2 (en) | Bot supervision |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21790696 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 3192505 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2023516792 Country of ref document: JP Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2021343437 Country of ref document: AU Date of ref document: 20210915 Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2021790696 Country of ref document: EP Effective date: 20230417 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202180076640.9 Country of ref document: CN |