WO2019113122A1 - Systèmes et procédés d'apprentissage automatique amélioré pour des conversations - Google Patents

Systèmes et procédés d'apprentissage automatique amélioré pour des conversations Download PDF

Info

Publication number
WO2019113122A1
WO2019113122A1 PCT/US2018/063928 US2018063928W WO2019113122A1 WO 2019113122 A1 WO2019113122 A1 WO 2019113122A1 US 2018063928 W US2018063928 W US 2018063928W WO 2019113122 A1 WO2019113122 A1 WO 2019113122A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
assistant
models
response
message
Prior art date
Application number
PCT/US2018/063928
Other languages
English (en)
Inventor
George Alexis Terry
Werner KOEPF
Siddhartha Reddy JONNALAGADDA
James D. HARRIGER
William Dominic WEBB-PURKIS
Macgregor S. GAINOR
Collin C. FERGUSON
Ravi Shankar
Shashi SHANKAR
Ian MCCANN
Keith GODFREY
Christopher Allan LONG
Brian Matthew KAMINSKI
John SANSONE
Jennifer KIRKLAND
Ryan Francis GINSTROM
Caleb Andrew BREDLOW
Kyle SARGENT
Alexander Carmelo Reid FORDYCE
Original Assignee
Conversica, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/208,484 external-priority patent/US11663409B2/en
Priority claimed from US16/208,488 external-priority patent/US20190179903A1/en
Priority claimed from US16/208,478 external-priority patent/US20190180196A1/en
Application filed by Conversica, Inc. filed Critical Conversica, Inc.
Publication of WO2019113122A1 publication Critical patent/WO2019113122A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G06F40/35—Discourse or dialogue representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

  • the present invention relates to systems and methods for innovative advances and applications in the generation and automatic models using statistical techniques including but not limited to machine learning, active learning, reinforcement learning, transfer learning, and deep learning.
  • the said models are applied for a variety of applications in conversational artificial intelligence (AI) including but not limited to message response generation, AI assistant performance, and other language processing, primarily in the context of the generation and management of a dynamic conversations.
  • AI conversational artificial intelligence
  • Such systems and methods provide a wide range of business people more efficient tools for outreach, knowledge delivery, automated task completion, and also improve computer functioning as it relates to processing documents for meaning.
  • such system and methods enable more productive business conversations and other activities with a majority of tasks performed previously by human workers delegated to artificial intelligence assistants.
  • AI Artificial Intelligence
  • AI tools examples include stock managers, chatbots, and voice activated search-based assistants such as Siri and Alexa.
  • the model updating may reuse existing business conversations as the training data set.
  • Features within the dataset may be defined and extracted.
  • Models may be selected and parameters for the models defined. Within a distributed computing setting the parameters may be optimized, and the models deployed.
  • the training data may be augmented over time to improve the models.
  • Visualization metrics for the models may also be generated and displayed. These visualization metrics may include accuracy, precision, recall, fl -score, and f_beta-score.
  • the visualization metrics may include generating a tree visualizer, response browser and an accuracy browser
  • Existing business conversations may be reused by manually identifying actions applicable to the conversations, automatically identifying context of responses in the conversation, generating instance-label pairs for each response, and randomly selecting a preset number of instance-label pairs as the test data set.
  • the defining and extracting features may include processing messages in the test data into sentences, parts of speech, normalized tokens, phrase chunks, syntactic dependencies, and constituency trees.
  • Next name entity recognition is performed to extract concepts.
  • the name entities may be normalized, and concept associations may be extracted.
  • a lexicon for the concept associations is generated from which the features are obtained.
  • Model deployment may leverage a docker which the model is inserted into.
  • a decision tree is generated using the docked model, and the model may be linked to a classifier service. Rules are added to assist the classifier service, and a server/network is then provisioned for the model.
  • the models may be versioned, and each version may be compared against prior versions to confirm improvement in model performance. Additionally, thresholds for model performance may be set allowing for fallback to hardrule systems or human intervention when required.
  • deep learning models may be employed to improve system accuracy. These deep learning models may be generated by collecting a corpus of human-to- human conversations, processing the conversations to remove boilerplate language, replacing entities in the processed conversations, converting the entity replaced conversations format to context, utterance and label, embedding the converted conversations, and convoluting the embedded conversations a number of times. The convoluting includes multiple sets of leamable filters with small receptive fields. The output of the convolution layers may be flattened, and rectifying linear units may be generated and max pooled. This results in a deep learning output that may then be combined with more traditional machine learning models to generate a hybrid model.
  • This deep learning methodology may employ convolutional neural networks, and in particular character level convolutional neural networks. Word2Vec and Glove and/or InferSent embedding may be leveraged with the convolutional neural networks.
  • the deep learning output is generated using bidirectional long short term memory (LSTM) encoders.
  • LSTM long short term memory
  • Active learning techniques may be employed for the generation of some models. Active learning may include uploading a preset number of sentences, suggesting high priority annotations in the uploaded sentences, generating multiple annotations per action, intent, or entity found in the uploaded sentences, selecting from the multiple annotations a subset of reliable annotations, where the subset is selected based upon low inter-annotator agreement, and building a machine learning model using the subset of reliable annotations.
  • the f-score for the model is calculated and compared to an acceptable level, which may be 95% in some cases. If below this threshold the system may repeat the process of training to improve the model performance.
  • the models developed and updated may be employed by a response system generally, or may function to enable specific types of AI systems.
  • One such a system may be an AI assistant that is designed to take use cases and objectives, and execute tasks until the objectives are met.
  • These AI systems are thus“rewards based” and may have access to a suite of external third party systems (such as calendar systems, frequently asked questions with approved answers, contact and CRM systems, etc.) as well as persisting memories of actions taken with various targets/leads in order to accomplish their objectives.
  • the objectives are initially selected for the AI assistant (often relating to a use case) and subsequently the resources, including access to third party systems, is determined based upon the objectives needing to be met.
  • the AI assistant engages in multiple rounds of conversations with the given target/lead using any of the previously discussed modeling methods to classify the conversations and take appropriate actions. These iterative conversations may continue until the particular objectives are met.
  • the AI assistant may include a marketing assistant, a customer service assistant, a customer success assistant, a recruiting assistant, a legal assistant, a finance assistant, a human resources assistant, a sales assistant, a social media assistant, and a focus group assistant.
  • the use cases may include handling inbound leads, handling aged leads, pre-event management, post-event management, outreach, and alternate contact; and the objectives for this assistant may include setting up appointments with a sales representative, beginning a nurturing conversation, and collecting new leads.
  • the use cases for a customer success assistant include expanding usage, renewal of a deal, winning back lost customers, advocate management, health checks, and events; while the objectives may include scheduling a call with a customer success manager, adoption of a feature, contract renewal, gathering feedback from customers, driving positive reviews, gathering feedback for product improvement, increasing customer usage, and driving event attendance.
  • For the finance assistant use cases may include collections, payment reminders and updating billing information, and objectives may include scheduling a call with accounts receivable, collecting payment prior to collections, and updating payment information.
  • For a recruiting assistant use cases may include candidate sourcing, applicant follow-up, and applicant pool interest, while objectives may include scheduling a call with a hiring manager or recruiter, generating summaries of candidates resume and virtual screen, salary negotiation, and support candidate with hiring paperwork.
  • For a human resources assistant use cases may include onboarding, orientation, employee support and employee satisfaction, and objectives may include providing documentation to employees responsive to needs, providing access to frequently asked questions with approved answers, satisfaction surveying, support candidate with hiring paperwork, benefits enrollment, and training.
  • For a legal assistant the use cases may include advice and investor relationships, and objectives include providing access to frequently asked questions with approved answers related to legal matters, collecting investor feedback, and scheduling meetings with corporate counsel.
  • Another system capable of leveraging the models includes an automated question answering system utilizing approved answers.
  • Such a system receives a response message from a human contact, identifies questions within the received response message using machine learning classifiers, cross references the identified questions with approved answer database, and outputs an approved answer from the approved answer database when there is a match. If no match is found a canned answer may be sent out instead. The outputs may be sent to a chatbot for display back to the user. Identifying the question may include identifying if a question is present and classifying the topic of the question. The topic of the question is then used for the cross reference against answers by topic. The answer topics and approved answers are provided by a third party company.
  • Yet another system for utilizing these various classification models is an intent based classification system for action determination.
  • Such a system allows mapping intents to actions using rules. Outputs of such a mapping are then received as examples in the form of text and an appropriate action in response to such a text. These outputs are used to generate a machine learning intent model. A response is then received, and the intent of the response is determined using the intent model. Deep learning models may be employed to extract entity information from the response as well. The intent and entity information is then used by an action model to determine the appropriate action to be taken for the response.
  • the action model may be developed, in some cases, using active learning techniques described above.
  • any of the above systems may be further enhanced by enabling multiple language analysis.
  • the present systems leverage dictionaries for all supported languages, and translations to reduce the needed level of training sets.
  • a primary language is selected and a full training set is used to build a model for the classification using this language.
  • Smaller training sets for the additional languages may be added into the machine learned model. These smaller sets may be less than half the size of a full training set, or even an order of magnitude smaller.
  • a response may be translated into all the supported languages, and this concatenation of the response may be processed for classification.
  • such systems may be capable of altering the language in which new messages are generated. For example, if the system detects that a response is in French, the classification of the response may be performed in the above mentioned manner, and similarly any additional messaging with this contact may be performed in French.
  • Figure 1 is an example logical diagram of a system for generation and implementation of messaging conversations, in accordance with some embodiment
  • Figure 2 is an example logical diagram of a dynamic messaging server, in accordance with some embodiment
  • Figure 3 is an example logical diagram of a user interface within the dynamic messaging server, in accordance with some embodiment
  • Figure 4 is an example logical diagram of a message generator within the dynamic messaging server, in accordance with some embodiment
  • Figure 5A is an example logical diagram of a message response system within the dynamic messaging server, in accordance with some embodiment
  • Figure 5B is an example logical diagram of a model trainer within the message response system, in accordance with some embodiment
  • Figure 5C is an example logical diagram of a learning system within the message response system, in accordance with some embodiment
  • Figure 5D is an example logical diagram of a intent based action decision engine within the message response system, in accordance with some embodiment
  • Figure 6 is an example flow diagram for a dynamic message conversation, in accordance with some embodiment
  • Figure 7 is an example flow diagram for the process of on-boarding a business actor, in accordance with some embodiment
  • Figure 8 is an example flow diagram for the process of building a business activity such as conversation, in accordance with some embodiment
  • Figure 9 is an example flow diagram for the process of generating message templates, in accordance with some embodiment.
  • Figure 10 is an example flow diagram for the process of implementing the conversation, in accordance with some embodiment
  • Figure 11 is an example flow diagram for the process of preparing and sending the outgoing message, in accordance with some embodiment
  • Figure 12 is an example flow diagram for the process of processing received responses, in accordance with some embodiment
  • Figure 13 is an example flow diagram for the process of document cleaning, in accordance with some embodiment
  • Figure 14 is an example flow diagram for the process automated model learning and updating, in accordance with some embodiment
  • Figure 15 is an example flow diagram for the process of reusing conversations as training data, in accordance with some embodiment.
  • Figure 16 is an example flow diagram for the process of feature definition and extraction, in accordance with some embodiment;
  • Figure 17 is an example flow diagram for the process of visualizing metrics, in accordance with some embodiment.
  • Figure 18 is an example flow diagram for the process of model deployment, in accordance with some embodiment.
  • Figure 19 is an example flow diagram for the process of training data augmentation, in accordance with some embodiment.
  • Figure 20 is an example flow diagram for the process of configuring models for human loop-in, in accordance with some embodiment
  • Figure 21 is an example illustration of a tree visualization, in accordance with some embodiment.
  • Figure 22 is an example illustration of a browser response visualization, in accordance with some embodiment
  • Figure 23 is an example illustration of an action accuracy visualization, in accordance with some embodiment.
  • Figure 24A is an example flow diagram for the process of generating deep learning hybrid models, in accordance with some embodiment
  • Figure 24B is a chart of model accuracies, in accordance with some embodiment.
  • Figure 25A is an example flow diagram for the process of intent based action response using a deep learning model, in accordance with some embodiment
  • Figure 25B is an example decision tree for intent based action for continuing messaging, in accordance with some embodiment
  • Figure 25C is an example decision tree for intent based action for taking an additional action, in accordance with some embodiment
  • Figure 25D is an illustration of an example accuracy chart for intent based action versus standard action processes, in accordance with some embodiment
  • Figure 26 is an example flow diagram for the process of model training leveraging active learning, in accordance with some embodiment
  • Figure 27A is an example flow diagram for the process of responding to frequent questions using approved answers, in accordance with some embodiment
  • Figure 27B is an example illustration of a screenshot of a conversation between a human and an AI system employing frequent questions using approved answers, in accordance with some embodiment
  • Figure 28A is an example flow diagram for the process of utilizing objective based AI assistants, in accordance with some embodiment
  • Figure 28B is an example flow diagram for the process of generating objective based AI assistants, in accordance with some embodiment
  • Figure 28C is an illustration of example specifications for a marketing assistant, in accordance with some embodiment.
  • Figure 28D is an illustration of example specifications for a customer success assistant, in accordance with some embodiment.
  • Figure 28E is an illustration of example specifications for a finance assistant, in accordance with some embodiment.
  • Figure 28F is an illustration of example specifications for a recruiting assistant, in accordance with some embodiment.
  • Figure 29 is an example flow diagram for the process of message routing, in accordance with some embodiment.
  • Figure 30 is an example flow diagram for the process of modeling using multiple languages, in accordance with some embodiment.
  • Figures 31A and 31B are example illustrations of a computer system capable of embodying the current invention.
  • the present invention relates to cooperation between business actors such as human operators and AI systems. While such systems and methods may be utilized with any AI system, such cooperation systems particularly excel in AI systems relating to the generation of automated messaging for business conversations such as marketing and other sales functions. While the following disclosure is applicable for other combinations, we will focus upon mechanisms of cooperation between human operators and AI marketing systems as an example, to demonstrate the context within which the cooperation system excels.
  • the following systems and methods are for improvements in AI model generation and utilization within conversation systems and for employment with assistant systems.
  • the goal of the message conversations is to enable a logical dialog exchange with a recipient, where the recipient is not necessarily aware that they are communicating with an automated machine as opposed to a human user. This may be most efficiently performed via a written dialog, such as email, text messaging, chat, etc. However, it is entirely possible that given advancement in audio and video processing, it may be entirely possible to have the dialog include audio or video components as well.
  • an AI system is employed within an AI platform within the messaging system to process the responses and generate conclusions regarding the exchange. These conclusions include calculating the context of a document, intents, entities, sentiment and confidence for the conclusions.
  • FIG. 1 is an example logical diagram of a system for generating and implementing messaging conversations, shown generally at 100.
  • several users l02a-n are illustrated engaging a dynamic messaging system 108 via a network 106.
  • messaging conversations may be uniquely customized by each user l02a-n in some embodiments.
  • users may be part of collaborative sales departments (or other collaborative group) and may all have common access to the messaging conversations.
  • the users l02a-n may access the network from any number of suitable devices, such as laptop and desktop computers, work stations, mobile devices, media centers, etc.
  • the network 106 most typically includes the internet, but may also include other networks such as a corporate WAN, cellular network, corporate local area network, or combination thereof, for example.
  • the messaging server 108 may distribute the generated messages to the various message delivery platforms 112 for delivery to the individual recipients.
  • the message delivery platforms 112 may include any suitable messaging platform. Much of the present disclosure will focus on email messaging, and in such embodiments the message delivery platforms 112 may include email servers (Gmail, yahoo, Hotmail, etc.). However, it should be realized that the presently disclosed systems for messaging are not necessarily limited to email messaging. Indeed, any messaging type is possible under some embodiments of the present messaging system. Thus, the message delivery platforms 112 could easily include a social network interface, instant messaging system, text messaging (SMS) platforms, or even audio telecommunications systems.
  • SMS text messaging
  • One or more data sources 110 may be available to the messaging server 108 to provide user specific information, message template data, knowledge sets, insights, and lead information. These data sources may be internal sources for the system’s utilization, or may include external third-party data sources (such as business information belonging to a customer for whom the conversation is being generated). These information types will be described in greater detail below.
  • FIG. 2 provides a more detailed view of the dynamic messaging server 108, in accordance with some embodiment.
  • the server is comprised of three main logical subsystems: a user interface 210, a message generator 220, and a message response system 230.
  • the user interface 210 may be utilized to access the message generator 220 and the message response system 230 to set up messaging conversations, and manage those conversations throughout their life cycle.
  • the user interface 210 includes APIs to allow a user’s device to access these subsystems.
  • the user interface 210 may include web accessible messaging creation and management tools, as will be explored below in some of the accompanying example screenshots.
  • FIG. 3 provides a more detailed illustration of the user interface 210.
  • the user interface 210 includes a series of modules to enable the previously mentioned functions to be carried out in the message generator 220 and the message response system 230. These modules include a conversation builder 310, a conversation manager 320 an AI manager 330, an insight manager 340, and a knowledge base manager 350.
  • the conversation builder 310 allows the user to define a conversation, and input message templates for each series within the conversation.
  • a knowledge set and lead data may be associated with the conversation to allow the system to automatically effectuate the conversation once built.
  • Lead data includes all the information collected on the intended recipients, and the knowledge set includes a database from which the AI can infer context and perform classifications on the responses received from the recipients.
  • the conversation manager 320 provides activity information, status, and logs of the conversation once it has been implemented. This allows the user l02a to keep track of the conversation’s progress, success and allows the user to manually intercede if required.
  • the conversation may likewise be edited or otherwise altered using the conversation manager 320.
  • the AI manager 330 allows the user to access the training of the artificial intelligence which analyzes responses received from a recipient.
  • One purpose of the given systems and methods is to allow very high throughput of message exchanges with the recipient with relatively minimal user input. To perform this correctly, natural language processing by the AI is required, and the AI (or multiple AI models) must be correctly trained to make the appropriate inferences and classifications of the response message.
  • the user may leverage the AI manager 330 to review documents the AI has processed and has made classifications for.
  • the insight manager 340 allows the user to manage insights.
  • insights are a collection of categories used to answer some question about a document. For example, a question for the document could include“is the lead looking to purchase a car in the next month?” Answering this question can have direct and significant importance to a car dealership. Certain categories that the AI system generates may be relevant toward the determination of this question. These categories are the‘insight’ to the question, and may be edited or newly created via the insight manager 340.
  • a knowledge set is set of tokens with their associated category weights used by an aspect (AI algorithm) during classification. For example, a category may include“continue contact?”, and associated knowledge set tokens could include statements such as“stop”,“do no contact”,“please respond” and the like.
  • the message generator 220 utilizes context knowledge 440 and lead data 450 to generate the initial message.
  • the message generator 220 includes a rule builder 410 which allows the user to define rules for the messages.
  • a rule creation interface which allows users to define a variable to check in a situation and then alter the data in a specific way. For example, when receiving the scores from the AI, if the insight is Interpretation and the chosen category is 'good', then have the Continue Messaging insight return 'continue'.
  • the rule builder 410 may provide possible phrases for the message based upon available lead data.
  • the message builder 420 incorporates those possible phrases into a message template, where variables are designated, to generate the outgoing message.
  • Specific phrases may be assigned category rankings related to various dimensions such as“formal vs. informal, education level, friendly tone vs. unfriendly tone, and other dimensions.” Additional category rankings for individual phrases may also be dynamically assigned based upon operational feedback in achieving conversational objectives so that more“successful” phrases may be more likely to be included in a particular message template. This is provided to the message sender 430 which formats the outgoing message and provides it to the messaging platforms for delivery to the appropriate recipient.
  • Figure 5A is an example logical diagram of the message response system 230.
  • the contextual knowledge base 440 is utilized in combination with response data 599 received from the person being messaged.
  • the message receiver 520 receives the response data 599 and provides it to the AI interface 510, objective modeler 530, and classifier engine 550 for feedback.
  • the AI interface 510 allows the AI platform (or multiple AI models) to process the response for context, insights, sentiments and associated confidence scores.
  • the classification engine 550 includes a suite of tools that enable better classification of the messages using models that have been automatically generated and updated by a model trainer 560. Based on the classifications generated by the AI and classification engine 550 tools lead objectives may be updated by the objective modeler 530.
  • the objective modeler may indicate what the objective to the next action in the conversation may entail.
  • the model trainer 560 is capable of using historical conversation histories to generate and improve classification models, as well as action response models for individual clients.
  • the model trainer utilizes iterative machine learning of training conversations. With each update iteration, accuracy of the models improves, reducing the need for human intervention or fallback to hard rules.
  • the learning systems 570 may be employed to improve model training accuracy and efficiency using deep learning and active learning techniques.
  • an intent based action decision engine 590 may utilize the received models and leverage intent based decision making to improve action accuracy over traditional machine learned or hard rule based decision making processes.
  • This component of the message response system may include a training data aggregation interface 561 which collects, or otherwise accesses, historical messaging exchanges. Generally these messaging exchanges (conversations) are between humans, thereby ensuring that the models are being trained to a“gold standard”. However, human- AI conversations, if properly vetted to ensure response accuracy, may likewise be employed as part of the training data.
  • the training data aggregator 561 may further include an interface where a user may manually identify actions that are applicable for a given conversation. For example, within a sales conversation setting, the user may identify within the conversation when various actions, such as continuing messaging, skip to follow-up, do not email, stop messaging and lead to review, for example, are applicable.
  • the data aggregator 561 may automatically segment the message responses by user context, not just of the present response, but also taking into consideration the messaging history across multiple communication channels.
  • context refers to time, location, language, individuals involved, and similar information.
  • the system may automatically process a response email into various sections, such as the body, subject, sender’s first and last name, sender’s email, and sent time.
  • the data aggregator 561 may then generate an instance-label pair for each response.
  • the instance is the various extracted context based upon the response, and the label corresponds to actions that were previously identified by the user.
  • the instance may be the email response and its individual sections such body and subject and this may be paired with one of the actions previously noted by the user, such as discontinuing messaging.
  • the data aggregator 561 next randomly selects and removes a portion of the data, and used this extracted portion as a test set.
  • the portion removed may be set to a default of 1000 instance-label pairings.
  • instance-label pairs may be selected as a test set. Larger sets yield more accuracy at a cost of processing overhead and cost of data extraction, transformation and loading from the human-human conversations.
  • the feature definition module 562 may process the body of each response located in the test set into sentences. This sentence processing may leverage regular expressions and machine learning algorithms for sentence boundary detection. Due to the propensity for conversation messaging to be“sloppy” with grammar and proper sentence structure, simple rule-based systems for determining sentence boundaries, such as those employed by a grammar checker, may often be insufficient. As such, machine learning based sentence boundary detection may be employed in some cases with superior results. [0094] The feature definition module 562 also tokens the responses using regular expressions, and tags parts of speech. Part of speech tagging may employ statistical sequential labeling algorithms. The tokens may be normalized using stemming
  • phrase chunks may be generated. These phrase chunks may include noun phrases, verb phrases, etc. through the usage of shallow parsing. Syntactic dependencies and constituency trees may be built using probabilistic context free grammar and deep learning. Deep learning may leverage character level convolutional neural networks, in some embodiments, and syntax net algorithms in other embodiments. Specific examples of implementation of deep learning will be provided in considerable details below.
  • the feature definition module 562 may also perform name entity recognition (NER) to extract concepts related to the business being discussed. Examples of this could include a person, for example. Concepts are extracted which are relevant to the actions associated with the response. In some embodiments, concepts in NER are identified using graph based and deep learning statistical sequential labeling algorithms. Examples of which include
  • CRF Conditional Random Fields
  • LSTM Bidirectional Long Short Term Memory
  • the feature definition module 562 also normalizes the named entities to canonical names and identifiers. This normalization may leverage database-based similarity and unsupervised machine learning measures. Associations may also be extracted between the concepts in the conversation using instance-based classification algorithms. For example, a PERSON liking a PRODUCT would be an association that can be determined between these two concepts.
  • the feature definition module 562 next develops a lexicon related to attributes of concepts and associations. For example, confirmation, declination, negation,
  • Rules may be applied to determine if the lexicon values are present, either fully or partially, in relevant discourse elements in each sentence being analyzed.
  • features may be obtained by the feature definition module 562 by combining and permuting the individual outputs of the above steps such as normalized tokens, phrase chunks, syntactical dependencies, normalized NER concepts, associations and the matches with lexicons.
  • the features are transformed including, but are not limited to weighting higher the tokens that appear multiple time in the instance higher, weighting lower the tokens that are not unique as determined by their statistical proportions, stripping or normalizing accents, ignoring the decoding errors using various criteria, converting to lower case, removing words in a lexicon file that are deemed to be unimportant, combining adjacent tokens in the feature vector in groups of two or more, ignoring those tokens that appear in too many instances or appear in too few instances, regularizing the feature vector to penalize for overfitting for using too many features, etc.
  • the parameter management module 563 defines all algorithms and corresponding parameters that will be tested for action classification along the various features. Algorithms that may be employed by the parameter management module 563 include K-neighbor classifier, support vector machines, Gaussian Process classifier, decision tree classifier, random forest classifier, multi-layer perceptron classifier, Ada Boost Classifier, Gaussian naive bayes, Quadratic Discriminant Analysis, Linear Discriminant Analysis, stochastic gradient descent classifier, Bagging Classifier, extra trees classifier, gradient boosting classifier and voting classifier.
  • K-neighbor classifier support vector machines
  • Gaussian Process classifier decision tree classifier
  • random forest classifier random forest classifier
  • multi-layer perceptron classifier Ada Boost Classifier
  • Gaussian naive bayes Gaussian naive bayes
  • Quadratic Discriminant Analysis Linear Discriminant Analysis
  • stochastic gradient descent classifier Bagging Classifier
  • extra trees classifier extra trees classifier
  • the parameter management module 563 may optimize them in a distributed computing setting. This may include performing an exhaustive search over the specified parameter values for an estimator.
  • Grid search cross validation or equivalent algorithm, may be employed for this estimator.
  • Grid search cross validation utilizes a“fit” and“score” method, and also implements a“predict”,“predict probability”, “decision function”,“transform” and“inverse transform” if implemented in the estimator.
  • the parameters of the estimator used to apply these methods are optimized by cross-validated grid-search over a parameter grid.
  • Alternate methods for optimizing the parameters may include using other estimators, such as Randomized Search cross validation or Sequential Nested Search cross validation.
  • Sequential Nested Search cross validation may be implemented locally, and may identify parameters that are independent. These may be sorted by order of importance and grid search or randomized grid search is performed only in individual groups of dependent parameters. This optimization results in minimizing computational time for optimal features, algorithms and their corresponding parameters, in that order.
  • all component features are extracted using map-reduce framework, the combination set of the component features is optimized, the top performing machine learning algorithms are optimized along with their parameters, and lastly the best ensemble of top-performing machine learning algorithms are optimized.
  • the metric visualization module 564 generates visualizations such as accuracy, precision, recall, fl -score and f_beta-score for the individual classifiers.
  • a tree visualizer illustrates classification trees by volumes, and allows a user to click on a tree to see information about the tree, such as total responses classified by the tree, total confident responses, distribution of the confident classifications, a confusion matrix and pure accuracy.
  • the confusion matrix lists the AI decision along the columns, and human decisions along the rows. Ideally, the matrix should have a high degree of agreement between rows and column, however trends where the AI miss-classifies the message may be determined by patterns in the matrix.
  • An example of a Confusion matrix is provided below:
  • Metrics may be calculated on a periodic basis, for example weekly, based upon the prior period’s validation set.
  • a response browser and action accuracy browser may likewise be generated for display to the user. Examples of the tree visualizer, response browser and action accuracy browser may be seen in relation to Figures 21, 22 and 23, respectively.
  • the tree identifier is seen listed at 2110 with an illustration of the classification tree.
  • a summary for the tree is provided at 2120, along with response set association 2130.
  • the tree efficacy is presented at 2140 for a validation date range.
  • the browser response display 2200 provides the user the ability to filter the responses by a number of features, including message client, conversation type, action taken, date range message series and industry, as seen at 2210. After selecting filters, the report of actions may be run as illustrated at 2220. The applicable responses are then displayed to the user.
  • the action accuracy browser display 2300 provides the user the ability to filter reports aby reviewer, system, industry, client date ranges, conversation type and message series, as seen at 2310.
  • a legend 2320 provides the user information regarding the report labels.
  • the resulting report 2330 illustrates the actions taken that match the filters selected, including the number of times the action was taken in aggregate, action taken by the human versus and computer, and differences in these actions decisions indicating true positives, false positives, true negatives and false negatives. Precision, recall, a d f-beta scores are also displayed for the actions.
  • the model deploy er 565 embeds the classification model that has been optimized into a docker image.
  • the docker image includes a REST API that exposes the model’s functionality and other diagnostic information (e.g., version number, etc.).
  • a decision tree is generated, which may include thresholds for determining when a classification is determined to be sufficiently confident to move forward.
  • the model is then linked to a classifier service which allows the addition of rules for which responses should be classified by the model. For example, the model may determine that there is interest in a product, but depending upon the rules different action responses may be made. For example, in one system the response may include setting up a meeting with a sales representative, while in another system the pricing information may be conveyed automatically through the same medium messaging has taken place in.
  • servers and network infrastructure is automatically provisioned for the new model.
  • This provisioning may utilize Kubemetes, or similar container orchestration system.
  • the model trainer 560 may also include functionality for hardrule fallback when a confidence threshold is not met by the deployed model.
  • the training data augmenter 566 operates after model deployment. After new responses are generated from existing and new client sources, the system may annotate all new responses. Any classification that the system is not sufficiently confident in, or classifications that are flagged by an active learning approach (as will be disclosed in greater detail below) are then collected. These collected classifications contribute to an additional instance-label set that is processed much like the initially determined instance-label pairings.
  • a model update module 567 uses these new instance-label pairs to augment the earlier pairs after a feature extraction process as described previously. This causes the training data to be fleshed out with additional data specifically chosen due to its classification difficulty or as suggested by the active learning.
  • a threshold for training set size versioning is selected. In some embodiments, this may be set to 10% increase in training set sizing. After the model is subjected to a training set that meets this threshold, the model may be saved as a separate version, allowing for comparison between versions and, if necessary, reversion to an earlier state if the training data is somehow corrupted.
  • the model update module 567 may compare the new model version against the previous model version for key metrics like accuracy, precision, higher recall, lower false positives, and lower false negatives. This comparison may utilize a randomized data set, or may utilize the original training set used for the initial model build for consistency in results. If the updated model is found to be superior to the earlier versions, then the system may build the model binary, embed it in a docker image and verify the docker embedded model matches the most recent versioned model. This verification may be performed by comparing outputs from the docker embedded model against a known set of outputs for a given set of inputs.
  • Separate thresholds may also be determined for the parameter optimizations performed previously.
  • Active learning strategies may be employed to efficiently determine which classifications do not meet the set thresholds, and therefore are best determined by human intervention. These algorithms used to determine which classifications should be handled by human may include uncertainty sampling algorithms, query by committee, expected model change, expected error reduction, variance reduction, balance exploration and exploitation, and exponentiated gradient exploration for active learning. These various methods shall be described in more detail below.
  • Figure 5C provides an example diagram of the learning systems 570 that include deep learning systems 571 and active learning systems 577.
  • the deep learning system 571 includes additional components that collect testing conversations 572, clean and perform entity replacement 573, format the response to utterance, context and label 574, and then embed the output and process it through multiple convolution filter layers and pooling layers 575 to generate deep learning outputs. These outputs may then be combined with traditional machine learning models via a hybridizer 576.
  • the active learning system 577 utilizes a sentence uploader 578 for collecting training data. Annotations are prioritized for the training data 579, and these prioritized annotations are received 580 from a human operator.
  • FIG. 5D components of the intent based decision engine 590 are shown.
  • This system leverages the models previously generated to perform intent based decision making that generally out performs traditional action response engines.
  • This system starts with an action rule engine 591 that allows a user to manually set up rules for actions based upon classification outputs.
  • An intent model is then generated by the intent model builder 592, which may leverage any of the previously discussed modeling components.
  • Responses are then received 593, and an intent model engine 594 is applied to determine the intention behind the response. Entities are also determined by a determiner 595 using models for entity extraction.
  • An action modeler 596 uses the intention information, in conjunction with the entity information, to determine an appropriate action for the response.
  • FIG. 6 an example flow diagram for a dynamic message conversation is provided, shown generally at 600.
  • the process can be broadly broken down into three portions: the on-boarding of a user (at 610), conversation generation (at 620) and
  • Figure 7 provides a more detailed look into the on-boarding process, shown generally at 610.
  • a user is provided (or generates) a set of authentication credentials (at 710). This enables subsequent authentication of the user by any known methods of authentication. This may include username and password combinations, biometric identification, device credentials, etc.
  • the lead data associated with the user is imported, or otherwise aggregated, to provide the system with a lead database for message generation (at 720).
  • context knowledge data may be populated as it pertains to the user (at 730).
  • This may be as simple as selecting dashboard layouts, to configuring confidence thresholds required before alerting the user for manual intervention.
  • Figure 8 is the example flow diagram for the process of building a conversation, shown generally at 620.
  • the user initiates the new conversation by first describing it (at 810).
  • Conversation description includes providing a conversation name, description, industry selection, and service type.
  • the industry selection and service type may be utilized to ensure the proper knowledge sets are relied upon for the analysis of responses.
  • the message templates in the conversation are generated (at 820). If the series is populated (at 830), then the conversation is reviewed and submitted (at 840). Otherwise, the next message in the template is generated (at 820).
  • Figure 9 provides greater details of an example of this sub-process for generating message templates. Initially the user is queried if an existing conversation can be leveraged for templates, or whether a new template is desired (at 910).
  • the new message templates are generated by populating the templates with existing templates (at 920). The user is then afforded the opportunity to modify the message templates to better reflect the new conversation (at 930). Since the objectives of many conversations may be similar, the user will tend to generate a library of conversations and conversation fragments that may be reused, with or without modification, in some situations. Reusing conversations has time saving advantages, when it is possible.
  • the user may opt to write the message templates from scratch using the Conversation Editor (at 940).
  • the Conversation Editor When a message template is generated, the bulk of the message is written by the user, and variables are imported for regions of the message that will vary based upon the lead data. Successful messages are designed to elicit responses that are readily classified. Higher classification accuracy enables the system to operate longer without user interference, which increases conversation efficiency and user workload.
  • Figure 10 is an example flow diagram for the process of implementing the conversation, shown generally at 630.
  • the lead data is uploaded (at 1010).
  • Lead data may include any number of data types, but commonly includes lead names, contact information, date of contact, item the lead was interested in, etc.
  • Other data can include open comments that leads supplied to the lead provider, any items the lead may have to trade in, and the date the lead came into the lead provider's system.
  • Lead data is specific to the industry, and individual users may have unique data that may be employed.
  • An appropriate delay period is allowed to elapse (at 1020) before the message is prepared and sent out (at 1030).
  • the waiting period is important so that the lead does not feel overly pressured, nor the user appears overly eager. Additionally, this delay more accurately mimics a human correspondence (rather than an instantaneous automated message). Additionally, as the system progresses and leams, the delay period may be optimized by the cadence optimizer to be ideally suited for the given message, objective, industry involved, and actor receiving the message. This cadence optimization is described in greater detail later in this disclosure.
  • Figure 11 provides a more detailed example of the message preparation and output.
  • the message within the series is selected based upon which objectives are outstanding (at 1110).
  • the messages will be presented in a set order; however, if the objective for a particular lead has already been met for a given series, then another message may be more appropriate.
  • the recipient didn’t respond as expected, or not at all, it may be desirous to have alternate message templates to address the lead most effectively.
  • the lead data is parsed through, and matches for the variable fields in the message templates are populated (at 1120).
  • the populated message is output to the communication channel appropriate messaging platform (at 1130), which as previously discussed typically includes an email service, but may also include SMS services, instant messages, social networks, audio networks using telephony or speakers and microphone, or video communication devices or networks or the like.
  • the contact receiving the messages may be asked if he has a preferred channel of communication. If so, the channel selected may be utilized for all future communication with the contact.
  • communication may occur across multiple different communication channels based upon historical efficacy and/or user preference. For example, in some particular situations a contact may indicate a preference for email communication. However, historically, in this example, it has been found that objectives are met more frequently when telephone messages are utilized.
  • the system may be configured to initially use email messaging with the contact, and only if the contact becomes unresponsive is a phone call utilized to spur the conversation forward.
  • system may randomize the channel employed with a given contact, and over time adapt to utilize the channel that is found to be most effective for the given contact.
  • the process waits for a response (at 1040). If a response is not received (at 1050) the process determines if the wait has been timed out (at 1060). Allowing a lead to languish too long may result in missed opportunities; however, pestering the lead too frequently may have an adverse impact on the relationship. As such, this timeout period may be user defined and will typically depend on the communication channel. Often the timeout period varies substantially, for example for email communication the timeout period could vary from a few days to a week or more. For real-time chat communication channel implementations, the timeout period could be measured in seconds, and for voice or video communication channel implementations, the timeout could be measured in fractions of a second to seconds.
  • the system continues to wait for a response (at 1050). However, once sufficient time has passed without a response, it may be desirous to return to the delay period (at 1020) and send a follow-up message (at 1030). Often there will be available reminder templates designed for just such a circumstance.
  • the process may continue with the response being processed (at 1070).
  • This processing of the response is described in further detail in relation to Figure 12.
  • the response is initially received (at 1210) and the document may be cleaned (at 1220).
  • Document cleaning is described in greater detail in relation with Figure 13.
  • adapters may be utilized to extract information from the document for shepherding through the cleaning and classification pipelines. For example, for an email, adapters may exist for the subject and body of the response, often a number of elements need to be removed, including the original message, HTML encoding for HTML style responses, enforce UTF-8 encoding so as to get diacritics and other notation from other languages, and signatures so as to not confuse the AI. Only after all this removal process does the normalization process occur (at 1310) where characters and tokens are removed in order to reduce the complexity of the document without changing the intended classification.
  • lemmatization at 1320
  • name entity replacement at 1330
  • the creation of n-grams at 1340
  • sentence extraction at 1350
  • noun-phrase identification at 1360
  • extraction of out-of- office features and/or other named entity recognition at 1370.
  • Each of these steps may be considered a feature extraction of the document.
  • extractions have been combined in various ways, which results in an exponential increase in combinations as more features are desired.
  • the present method performs each feature extraction in discrete steps (on an atomic level) and the extractions can be“chained” as desired to extract a specific feature set.
  • the document is then provided to the AI platform for classification using the knowledge sets (at 1230).
  • the system initially applies natural language processing through one or more AI machine learning models to process the message for concepts contained within the message.
  • AI machine learning models there are a number of known algorithms that may be employed to categorize a given document, including Hardrule, Naive Bayes, Sentiment, neural nets including convolutional neural networks and recurrent neural networks and variations, k-nearest neighbor, other vector based algorithms, etc. to name a few.
  • the classification model may be automatically developed and updated as previously touched upon, and as described in considerable detail below as well. Classification models may leverage deep learning and active learning techniques as well, as will also be discussed in greater detail below.
  • the system renders insights from the message.
  • Insights are categories used to answer some underlying question related to the document.
  • the classifications may map to a given insight based upon the context of the conversation message.
  • a confidence score, and accuracy score, are then generated for the insight.
  • Insights are used by the model to generate actions.
  • Objectives of the conversation may be used to redefine the actions collected and scheduled. For example,‘skip-to-follow-up’ action may be replaced with an‘informational message’ introducing the sales rep before proceeding to‘series 3’ objectives. Additionally,‘Do Not Email’ or‘Stop Messaging’ classifications should deactivate a lead and remove scheduling at any time during a lead’s life-cycle. Insights and actions may also be annotated with“facts”. For example, if the determined action is to “check back later” this action may be annotated with a date‘fact’ that indicates when the action is to be implemented.
  • the actions received from the inference engine may be set (at 1240). A determination is made whether there is an action conflict (at 1250). Manual review may be needed when such a conflict exists (at 1270). Otherwise, the actions may be executed by the system (at 1260).
  • a determination is made whether to deactivate the lead (at 1075). Such a deactivation may be determined as needed when the lead requests it. If so, then the lead is deactivated (at 1090). If not, the process continues by determining if the conversation for the given lead is complete (at 1080). The conversation may be completed when all objectives for the lead have been met, or when there are no longer messages in the series that are applicable to the given lead. Once the conversation is completed, the lead may likewise be deactivated (at 1090).
  • the process may return to the delay period (at 1020) before preparing and sending out the next message in the series (at 1030).
  • the process iterates in this manner until the lead requests deactivation, or until all objectives are met. This concludes the main process for a comprehensive messaging conversation. Attention will now be focused on processes for model generation and automatic updating, deep learning, active learning, and usage of these models and methods for frequently asked questions with approved answers and AI assistants.
  • a process 1400 for automated model learning and updating is provided, in accordance with some embodiment.
  • conversations are reused as a source of training data (at 1410).
  • Figure 15 provides more detail into this step.
  • Actions applicable to the conversations are manually identified (at 1411).
  • the system automatically identifies the context of the response (at 1412).
  • “context” of the conversation art relevant attributes such as medium, time, sender information, etc.
  • Instance-label pairs for the responses are created (at 1413) as discussed previously.
  • the system randomly selects and removes a portion of the data as a test set (at 1414). In some embodiments, a default of 5000 instance-label pairs may be randomly selected as the training set.
  • FIG. 16 provides more detail into this step.
  • the message body for the training set are processed into sentences, parts of speech, normalized tokens, phrase chunks, syntactical dependencies and constituency trees (at 1421).
  • Name entity recognition is then performed (at 1422) for concept extraction.
  • the named entities are normalized (at 1423) and concept associations are extracted (at 1424).
  • a lexicon of related attributes of concepts and their associations is generated (at 1425) and features are obtained (at 1426) by combining and permuting the normalized name entities and concept associations.
  • the parameters of the model are defined (at 1430). As previously discussed, a number of classification algorithms may be utilized and for each algorithm, the parameters utilized can be identified. Subsequently, the parameters for each of these classification algorithms may be optimized for in a distributed computing environment (at 1440). Parameter optimization may utilize a number of searching algorithms as discussed previously, which extract component features, optimize the combination set of component feature, optimize the top-performing machine learning algorithms along their parameters and optimize the best ensemble of top-performing machine learning algorithms.
  • an accuracy visualization is generated (at 1451), as are a precision visualization (at 1452), a recall visualization (at 1453), a fl -score and f_beta score visualization (at 1454), and a conversica score visualization (at 1455).
  • These visualizations are populated into a tree visualizer interface (at 1456) as previously discussed.
  • a response browser may also be generated (at 1457) as previously discussed.
  • an example of such an interface may be seen in relation to Figure 22.
  • an action accuracy browser may also be generated (at 1458) as previously discussed.
  • an example of such an interface may be seen in relation to Figure 23.
  • the models may be deployed (at 1460), which is described in further detail in relation to the example process shown at Figure 18.
  • the model is embedded in a docker image (at 1461) and a decision tree utilizing the model outputs is generated (at 1462).
  • the model is linked to a classifier service (at 1463), and rules are added to the model to assist in classification (at 1464).
  • a server and network infrastructure are provisioned for the new model (at 1465).
  • a hard rule fallback process is performed (at 1470). This hard rule fallback process may be employed when the deployed model falls below a confidence threshold, but before a human intervention is required. In some cases, deficiency in the model may be adequately addressed utilizing more traditional hard rule processes, thereby enabling continued automated performance by the system.
  • training data may be augmented (at 1480), which is described in greater detail in relation to the example process of Figure 19.
  • feature extraction is repeated on newly received conversation responses (at 1481) to generate a new set of instance-label pairs for the new conversation data.
  • the existing training instance-label pairs may be augmented with these new instance- label pairs (at 1482).
  • the model version may be retained based upon a delta in training sample size (at 1483). For example, for every 10% increase in training sample set size, the model may be saved as an updated version.
  • These versions may be verified against earlier versions using known input-output pairs to determine model precision, accuracy, recall, false positive and false negative rates (at 1484). Only superior models are then used to build out a model binary and deployed using a docker image (at 1485).
  • a number of methodologies that identify problematic areas of the model may be employed for the determination of the categorical areas to be handled by humans, as previously discussed.
  • the conversations that are below the thresholds, and those falling within the determined categories, may be routed to a human operator (at 1493).
  • a classification model may be automatically generated and continually refined.
  • Such models are integral to the efficient operation of a conversational system, as discussed extensively, but may also have implications for the operation of more refined reward-based AI tools, as will be discussed below.
  • FIG. 24A is an example flow diagram for the process 2400 of generating deep learning hybrid models, in accordance with some embodiment.
  • Deep learning models as discussed previously, has implications for the feature extraction of the responses, particularly as it pertains to the development of the generation of constituency trees and name entity recognition.
  • a corpus of human-to-human conversations may be obtained (at 2410). These conversations may be processed to remove boilerplate and for entity replacement (at 2420). The format of the resulting conversations is then converted to context, utterance and labels (at 2430).
  • deep learning may leverage convolutional neural networks (CNN) with Word2Vec and Glove embedding.
  • character level CNN may be particularly effective.
  • Such systems include an embedding layer (at 2440), which may include InferSent Embeddings, followed by multiple layers of convolutions (at 2450) which consist of sets of leamable filters, each having a small receptive field, and passing forward to the subsequent convolution layer.
  • the output is pooled (at 2460) which is a non-linear down sampling.
  • Bidirectional Long Short Term Memory (LSTM) encoders with Max Pooling may be employed in some embodiments.
  • the fully connected layer (Max Pooling) produces the deep learning output (at 2480) which may be combined with traditional machine learning to generate an ensemble model (at 2490). For example, to combine Char CNN, stacking may be used to learn a meta classifier based on probability values of individual actions by the component models.
  • Algorithms that are chosen previously are diversified in nature and work well on different segments of our data-set so that the meta classifier will benefit from the respective strengths of each individual classifier.
  • the ensemble framework provides flexibility in terms of adding or removing models as they are loosely coupled with the meta classifier.
  • the ensemble of deep learning with traditional machine learning may be utilized for action classification and entity extraction, and may produce superior results as compared to merely using traditional machine learning techniques.
  • Figure 24B provides a chart 2400B of accuracy measures of different conversations using state of the art production models, agreement accuracy and deep learning accuracy. Note that while deep learning is not always the most accurate model, it often outperforms other models, and outperform the production model the majority of the time.
  • Figure 25 A is a flow diagram for the example process 2500 of intent based action response using the deep learning model ensemble, in accordance with some embodiment.
  • a response is mapped directly to an action.
  • intent based approach the response is mapped to entities and intents, which are structured computer interpretable representations of what the response actually means. The intent and entity information is then used to determine the actions, which over time has been shown to be more accurate than direct response to action modeling.
  • the client maps intentions to actions using a rule based system
  • the client also provides new examples and/or corrections to outputs of the rule based system for training in the machine learning system (at 2520).
  • the client monitors the intent model and continually provides mapping of actions to intent until satisfied (at 2530). Once the client is satisfied with the model’s performance a response is received by the system (at 2540), and the intent classification is performed using active learning and/or the automated model building discussed previously (at 2550). Deep learning is then used to tag entities in the response (at 2560) and the model is utilized to determine actions based on the intent and entities (at 2570).
  • Figures 25B and 25C provide examples of decision trees 2500B and 2500C, respectively, for such intent based decision processes.
  • the first decision tree 2500B is for determining whether to continue contacting an individual, whereas the second decision tree 2500C is for generating an action response.
  • Figure 25D provides a chart 2500D that illustrates the comparative accuracy of a standard machine learning based action response model, versus intent based response modeling and a combined model.
  • Intent alone is shown at 2501, and performs very well at the outset when there are very few training samples.
  • the combined system 2503 and standard machine learned system (2502)’ s accuracies improve dramatically and rapidly outpace intent alone modeling.
  • the combined system proves to be the best and most accurate modeling system. This information may be leveraged when making decisions and the size of the training sample is known, such that the model utilized can depend accordingly.
  • FIG. 26 is an example flow diagram for the process 2600 of model training leveraging active learning, as used in the intent based approach for intent classification.
  • Active learning as used in this application is the process whereby a preset number of sentences is initially uploaded (at 2610) and is used to suggest high priority annotations within the uploaded sentences (at 2620).
  • sequential approaches such as Active Thompson Sampling (ATS) and Exponentiated gradient (EG) may be used to determine annotations that are high priority.
  • ATS samples inputs after assigning a sampling distribution the data pool whereas EG applies optimal random exploration.
  • the preset number of sentences to be reviewed is set to one million, but any suitably large number of sentences may be utilized.
  • Figure 27A is an example flow diagram for the process 2700 of responding to frequent questions using approved answers by leveraging the automatically generated and updated classification models, in accordance with some embodiment.
  • the classification models have already been generated using any of the already discussed methods.
  • a response is received (at 2710) and through the classification it is determined that a question is present in the response (at 2720).
  • Topics for the questions are determined by the classifier.
  • the question topic is cross referenced against generic question topics for which approved answers have already been generated (at 2730).
  • the system may automatically output the answer or one of the semantically similar approved versions of the answer (at 2750), otherwise the system may output a canned response along the lines of“A representative will answer this question shortly” (at 2760). Regardless of output type, the output in this example system may be uploaded into a chat-bot for communication with the user (at 2770).
  • An example of such a conversation with a chat-bot is provided in the example screenshot 2700B shown in relation to Figure 27B.
  • FIG. 28A is an example flow diagram for the process 2800 of utilizing objective based AI assistants, in accordance with some embodiment.
  • a client user selects objectives for the AI assistant (at 2810) from a myriad of possible objectives.
  • the AI system is required to have access to the client’s third party systems related to completing these objectives (at 2820).
  • the AI assistant will typically require access to communication channel appropriate systems such as email systems, along with contact databases, calendar applications and other systems that contain data that’s appropriate and helpful for the use cases being addressed by the conversational AI assistant.
  • the assistant will automatically converse with leads to meet the objectives (at 2830) and take actions, when appropriate, to satisfy the objective conditions.
  • objectives may include obtaining particular information about something (e.g., determine a customer’s views, beliefs or opinions regarding a particular topic, etc.), classifying or scoring a lead into a category or metric, altering a target’s opinion or perspective on a topic, or mere information dissemination.
  • the AI assistants have the capacity to have a persistent memory of conversations, and may be enabled to have access to external data sources when coupled to appropriate third party systems. In some cases, the assistant may be enabled to have unlimited series within a given conversation until an objective is met, and may support multiple language models and multiple communication channel appropriate models and message templates.
  • the AI assistant may be designed for any task. However, within a business setting a few“prototypical” AI assistants emerge.
  • Figure 28B provides a flowchart for the process of generating these various assistants, shown generally at 2805. These include generating a marketing assistant 2815, a sales assistant 2825, a customer support assistant 2835, a recruiting assistant 2845, a finance assistant 2855, a legal assistant 2865, a human resources assistant 2875 and a customer success assistant 2885.
  • Additional assistants could also include a social media management assistant and a pricing assistant and could be created to automate virtually any routine business conversation.
  • Figure 28C provides an example specification chart 2800C for a marketing assistant.
  • This example marketing assistant may have a series of use cases tied to one or more of a set of objectives.
  • the objectives for a marketing assistant would typically include setting up an appointment with sales, lead collection, and nurturing a relationship with a customer.
  • the use cases may include following up with inbound leads, dealing with aged leads, pre-event management, post event management, outreach and determining alternative contacts, in this example.
  • Figure 28D provides an example specification chart 2800D for a customer success assistant.
  • This example assistant for a customer success department may have objectives tied to adoption of new product features, scheduling a call with a customer success manager, contract renewal, feedback gathering, etc.
  • Use cases for such an assistant could include upselling or expanding product or service usage, contract renewal, winning back lost customers, advocacy management, customer engagement to determine performance optics, and event management, for example.
  • these use cases and objectives may differ from that of a recruiter assistant which may be concerned with scheduling a call with a hiring manager as the only objective, and the use cases being candidate sourcing, applicant follow-up, and past applicant pool interest, as illustrated on the specification chart 2800F associated with Figure 28F.
  • a finance assistant on the other hand may have objectives to schedule a call with accounts receivable, acquire direct payment, and updating payment details. These may map directly to use cases of collections, payment reminders and updating billing information, as illustrated on the specification chart 2800E associated with Figure 28E.
  • One result of such a natural and organic conversation between a target and a given AI assistant is that the target will often become comfortable with interacting with the AI assistant, and may ask questions of the assistant that are outside of the expertise of the AI assistant.
  • This is basic human nature: once the target has the contact information for the AI assistant who has been helpful in the past , and has thus built a relationship with a particular AI assistant, the target individual is likely to reengage the same AI if he has additional questions or concerns.
  • the disclosed AI assistants are highly capable of communicating with a target within a given use case for a specific objective. However as the topics being conversed about deviate from these well-understood topics, the AI assistant may be the incorrect vehicle to continue the conversation with the target. In these situations, the AI assistants may employ message routing capabilities to ensure the human target of the conversation receives the correct answers and the best user experience possible.
  • Figure 29 provides an example flow diagram for a method for message routing, shown generally at 2900.
  • the human target provides a message to the AI assistant (at 2910) which when analyzed using the AI assistants classification models is not found to include a topic or classification that the AI assistant is capable of handling in an optimal manner.
  • the AI assistant at 2910
  • a separate generic classification model may be employed to classify the message topic (at 2920).
  • the result of this generic classification of the message may be cross referenced against an internal listing of expertise systems and/or individuals (at 2930). This listing of experts, the contact information associated with these experts, and the rules associating a classification to a given expert, may be maintained by a customer that is employing the AI assistant system.
  • the system may provide this contact information back to the target (at 2940), automatically forward the message to this expert (at 2950) or do both.
  • the human target is given the contact information of the system (such as another AI assistant), or individual (such as a sales representative), that is best able to address the needs of the target individual.
  • Figure 30 provides an example flowchart for the process of handling multiple languages in an improved manner over traditional methodologies, shown generally at 3000.
  • dictionaries are collected for the supported languages (at 3010).
  • the classification models may be trained using a central language and training datasets in alternate languages may be incorporated slowly over time (at 3020) to enable multi-lingual classifications.
  • the language employed is initially determined (at 3030).
  • the response is then translated into all available languages (at 3040) to allow for human operator audit and review. Classification may be performed on all response translations (at 3050), and confidence measures for each may be determined.
  • this classification may be utilized; however, if the native language classification is significantly lower than one of the translations, the classification with the highest confidence may alternatively be used, regardless of language employed.
  • n-gram and deep learning models may be employed on a concatenation of the multiple languages for the classification of the response.
  • the present systems and methods are capable of storing information regarding the language preferred by the contact, and may ensure that future communications with this contact are generated in this preferred language.
  • Figures 31 A and 31B illustrate a Computer System 3100, which is suitable for implementing embodiments of the present invention.
  • Figure 31A shows one possible physical form of the Computer System 3100.
  • the Computer System 3100 may have many physical forms ranging from a printed circuit board, an integrated circuit, and a small handheld device up to a huge super computer.
  • Computer system 3100 may include a Monitor 3102, a Display 3104, a Housing 3106, a Disk Drive 3108, a Keyboard 3110, and a Mouse 3112.
  • Disk 3114 is a computer-readable medium used to transfer data to and from Computer System 3100.
  • FIG. 31B is an example of a block diagram for Computer System 3100. Attached to System Bus 3120 are a wide variety of subsystems.
  • Processor(s) 3122 also referred to as central processing units, or CPUs
  • Memory 3124 includes random access memory (RAM) and read-only memory (ROM).
  • RAM random access memory
  • ROM read-only memory
  • RAM random access memory
  • ROM read-only memory
  • Both of these types of memories may include any suitable of the computer-readable media described below.
  • a Fixed Disk 3126 may also be coupled bi directionally to the Processor 3122; it provides additional data storage capacity and may also include any of the computer-readable media described below.
  • Fixed Disk 3126 may be used to store programs, data, and the like and is typically a secondary storage medium (such as a hard disk) that is slower than primary storage. It will be appreciated that the information retained within Fixed Disk 3126 may, in appropriate cases, be incorporated in standard fashion as virtual memory in Memory 3124.
  • Removable Disk 3114 may take the form of any of the computer-readable media described below.
  • Processor 3122 is also coupled to a variety of input/output devices, such as Display 3104, Keyboard 3110, Mouse 3112 and Speakers 3130.
  • an input/output device may be any of: video displays, track balls, mice, keyboards, microphones, touch- sensitive displays, transducer card readers, magnetic or paper tape readers, tablets, styluses, voice or handwriting recognizers, biometrics readers, motion sensors, brain wave readers, or other computers.
  • Processor 3122 optionally may be coupled to another computer or telecommunications network using Network Interface 3140. With such a Network Interface 3140, it is contemplated that the Processor 3122 might receive information from the network, or might output information to the network in the course of performing the above-described model learning and updating processes.
  • method embodiments of the present invention may execute solely upon Processor 3122 or may execute over a network such as the Internet in conjunction with a remote CPU that shares a portion of the processing.
  • a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as“implemented in a computer-readable medium.”
  • a processor is considered to be“configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
  • the computer system 3100 can be controlled by operating system software that includes a file management system, such as a disk operating system.
  • a file management system such as a disk operating system.
  • operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Washington, and their associated file management systems.
  • Windows® is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Washington, and their associated file management systems.
  • Windows® from Microsoft Corporation of Redmond, Washington
  • Windows® Windows® from Microsoft Corporation of Redmond, Washington
  • Linux operating system is the Linux operating system and its associated file management system.
  • the file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
  • the machine operates as a standalone device or may be connected (e.g., networked) to other machines.
  • the machine may operate in the capacity of a server or a client machine in a client-server network environment or as a peer machine in a peer-to-peer (or distributed) network environment.
  • the machine may be a server computer, a client computer, a virtual machine, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
  • PC personal computer
  • PDA personal digital assistant
  • machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term“machine-readable medium” and“machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • the term“machine- readable medium” and“machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.
  • routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as“computer programs.”
  • the computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Machine Translation (AREA)

Abstract

Cette invention concerne des systèmes et des procédés pour d'amélioration de l'apprentissage et de la mise à jour de modèles en intelligence artificielle. La mise à jour de modèles peut réutiliser des conversations commerciales existantes en tant qu'ensemble de données d'apprentissage. Des caractéristiques dans l'ensemble de données peuvent être définies et extraites. Des modèles peuvent être sélectionnés et des paramètres pour les modèles définis. Dans un environnement informatique distribué, les paramètres peuvent être optimisés, et les modèles déployés. Les données d'apprentissage peuvent être augmentées dans le temps pour améliorer les modèles. Des modèles d'apprentissage profond ainsi que des techniques d'apprentissage actif, peuvent être utilisés pour améliorer la précision du système. Les modèles développés et mis à jour peuvent être utilisés par un système de réponse de manière générale, ou peuvent fonctionner pour mettre en œuvre des types spécifiques de systèmes d'intelligence artificielle. Un tel système peut être un assistant d'intelligence artificielle qui est conçu pour utiliser des cas d'usage et des objectifs, et exécuter des tâches jusqu'à ce que les objectifs soient satisfaits. Un autre système capable d'exploiter les modèles comprend un système de réponse automatisé utilisant des réponses approuvées. Un autre système d'utilisation de ces différents modèles de classification est un système de classification basé sur l'intention pour une détermination d'action. Enfin, il est à noter que tous les systèmes ci-dessus peuvent être davantage améliorés en mettant en œuvre une analyse à langues multiples.
PCT/US2018/063928 2017-12-04 2018-12-04 Systèmes et procédés d'apprentissage automatique amélioré pour des conversations WO2019113122A1 (fr)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US201762594415P 2017-12-04 2017-12-04
US62/594,415 2017-12-04
US16/208,484 US11663409B2 (en) 2015-01-23 2018-12-03 Systems and methods for training machine learning models using active learning
US16/208,488 2018-12-03
US16/208,488 US20190179903A1 (en) 2015-01-23 2018-12-03 Systems and methods for multi language automated action response
US16/208,484 2018-12-03
US16/208,478 US20190180196A1 (en) 2015-01-23 2018-12-03 Systems and methods for generating and updating machine hybrid deep learning models
US16/208,478 2018-12-03

Publications (1)

Publication Number Publication Date
WO2019113122A1 true WO2019113122A1 (fr) 2019-06-13

Family

ID=66751765

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/063928 WO2019113122A1 (fr) 2017-12-04 2018-12-04 Systèmes et procédés d'apprentissage automatique amélioré pour des conversations

Country Status (1)

Country Link
WO (1) WO2019113122A1 (fr)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110728313A (zh) * 2019-09-29 2020-01-24 北京声智科技有限公司 一种用于意图分类识别的分类模型训练方法及装置
CN111241236A (zh) * 2019-12-30 2020-06-05 新大陆数字技术股份有限公司 任务导向性问答方法、系统、电子设备、可读存储介质
CN112131366A (zh) * 2020-09-23 2020-12-25 腾讯科技（深圳）有限公司 训练文本分类模型及文本分类的方法、装置及存储介质
CN112800764A (zh) * 2020-12-31 2021-05-14 江苏网进科技股份有限公司 一种基于Word2Vec-BiLSTM-CRF模型的法律领域的实体抽取方法
CN112825143A (zh) * 2019-11-20 2021-05-21 北京眼神智能科技有限公司 深度卷积神经网络压缩方法、装置、存储介质及设备
CN113050955A (zh) * 2019-12-26 2021-06-29 阿尔法云计算(深圳)有限公司 一种自适应ai模型部署方法
CN113434662A (zh) * 2021-06-24 2021-09-24 平安国际智慧城市科技股份有限公司 文本摘要生成方法、装置、设备和存储介质
CN113553097A (zh) * 2021-07-30 2021-10-26 上海玳鸽信息技术有限公司 模型版本管理方法及装置
WO2021231195A1 (fr) * 2020-05-13 2021-11-18 Factset Research System, Inc. Procédé et système de mappage d'étiquettes dans des tables normalisées à l'aide d'un apprentissage automatique
KR20210138266A (ko) * 2020-05-12 2021-11-19 인하대학교 산학협력단 딥러닝 기반 키워드 추출 방법 및 장치
CN114757307A (zh) * 2022-06-14 2022-07-15 中国电力科学研究院有限公司 一种人工智能自动训练方法、系统、装置及存储介质
US11436528B2 (en) 2019-08-16 2022-09-06 International Business Machines Corporation Intent classification distribution calibration
KR102456513B1 (ko) * 2022-03-04 2022-10-20 주식회사 테스트웍스 생성 모델을 이용한 데이터 증강 처리 시스템 및 그 방법
US11641330B2 (en) 2020-08-06 2023-05-02 International Business Machines Corporation Communication content tailoring
CN116451056A (zh) * 2023-06-13 2023-07-18 支付宝(杭州)信息技术有限公司 端特征洞察方法、装置以及设备
US11727923B2 (en) 2020-11-24 2023-08-15 Coinbase, Inc. System and method for virtual conversations
US11770307B2 (en) 2021-10-29 2023-09-26 T-Mobile Usa, Inc. Recommendation engine with machine learning for guided service management, such as for use with events related to telecommunications subscribers
EP4046088A4 (fr) * 2019-10-15 2023-10-18 UiPath, Inc. Utilisation d'intelligence artificielle pour la sélection et le chaînage de modèles pour l'automatisation de processus robotiques
CN117910458A (zh) * 2024-01-19 2024-04-19 上海算法创新研究院 一种基于大语言模型的文本数据清洗系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090228264A1 (en) * 2003-02-11 2009-09-10 Microsoft Corporation Management of conversations
US8346563B1 (en) * 2012-04-10 2013-01-01 Artificial Solutions Ltd. System and methods for delivering advanced natural language interaction applications
US20160232457A1 (en) * 2015-02-11 2016-08-11 Skytree, Inc. User Interface for Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions and Features
US20170091320A1 (en) * 2015-09-01 2017-03-30 Panjiva, Inc. Natural language processing for entity resolution
US20170323065A1 (en) * 2014-11-06 2017-11-09 Ieso Digital Health Limited Analysing text-based messages sent between patients and therapists

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090228264A1 (en) * 2003-02-11 2009-09-10 Microsoft Corporation Management of conversations
US8346563B1 (en) * 2012-04-10 2013-01-01 Artificial Solutions Ltd. System and methods for delivering advanced natural language interaction applications
US20170323065A1 (en) * 2014-11-06 2017-11-09 Ieso Digital Health Limited Analysing text-based messages sent between patients and therapists
US20160232457A1 (en) * 2015-02-11 2016-08-11 Skytree, Inc. User Interface for Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions and Features
US20170091320A1 (en) * 2015-09-01 2017-03-30 Panjiva, Inc. Natural language processing for entity resolution

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11436528B2 (en) 2019-08-16 2022-09-06 International Business Machines Corporation Intent classification distribution calibration
CN110728313A (zh) * 2019-09-29 2020-01-24 北京声智科技有限公司 一种用于意图分类识别的分类模型训练方法及装置
EP4046088A4 (fr) * 2019-10-15 2023-10-18 UiPath, Inc. Utilisation d'intelligence artificielle pour la sélection et le chaînage de modèles pour l'automatisation de processus robotiques
US11893371B2 (en) 2019-10-15 2024-02-06 UiPath, Inc. Using artificial intelligence to select and chain models for robotic process automation
CN112825143A (zh) * 2019-11-20 2021-05-21 北京眼神智能科技有限公司 深度卷积神经网络压缩方法、装置、存储介质及设备
CN113050955A (zh) * 2019-12-26 2021-06-29 阿尔法云计算(深圳)有限公司 一种自适应ai模型部署方法
CN111241236A (zh) * 2019-12-30 2020-06-05 新大陆数字技术股份有限公司 任务导向性问答方法、系统、电子设备、可读存储介质
CN111241236B (zh) * 2019-12-30 2023-08-22 新大陆数字技术股份有限公司 任务导向性问答方法、系统、电子设备、可读存储介质
KR102476383B1 (ko) 2020-05-12 2022-12-09 인하대학교 산학협력단 딥러닝 기반 키워드 추출 방법 및 장치
KR20210138266A (ko) * 2020-05-12 2021-11-19 인하대학교 산학협력단 딥러닝 기반 키워드 추출 방법 및 장치
US11900272B2 (en) 2020-05-13 2024-02-13 Factset Research System Inc. Method and system for mapping labels in standardized tables using machine learning
WO2021231195A1 (fr) * 2020-05-13 2021-11-18 Factset Research System, Inc. Procédé et système de mappage d'étiquettes dans des tables normalisées à l'aide d'un apprentissage automatique
US11641330B2 (en) 2020-08-06 2023-05-02 International Business Machines Corporation Communication content tailoring
CN112131366B (zh) * 2020-09-23 2024-02-09 腾讯科技（深圳）有限公司 训练文本分类模型及文本分类的方法、装置及存储介质
CN112131366A (zh) * 2020-09-23 2020-12-25 腾讯科技（深圳）有限公司 训练文本分类模型及文本分类的方法、装置及存储介质
US11727923B2 (en) 2020-11-24 2023-08-15 Coinbase, Inc. System and method for virtual conversations
CN112800764B (zh) * 2020-12-31 2023-07-04 江苏网进科技股份有限公司 一种基于Word2Vec-BiLSTM-CRF模型的法律领域的实体抽取方法
CN112800764A (zh) * 2020-12-31 2021-05-14 江苏网进科技股份有限公司 一种基于Word2Vec-BiLSTM-CRF模型的法律领域的实体抽取方法
CN113434662B (zh) * 2021-06-24 2022-06-24 平安国际智慧城市科技股份有限公司 文本摘要生成方法、装置、设备和存储介质
CN113434662A (zh) * 2021-06-24 2021-09-24 平安国际智慧城市科技股份有限公司 文本摘要生成方法、装置、设备和存储介质
CN113553097A (zh) * 2021-07-30 2021-10-26 上海玳鸽信息技术有限公司 模型版本管理方法及装置
US11770307B2 (en) 2021-10-29 2023-09-26 T-Mobile Usa, Inc. Recommendation engine with machine learning for guided service management, such as for use with events related to telecommunications subscribers
KR102456513B1 (ko) * 2022-03-04 2022-10-20 주식회사 테스트웍스 생성 모델을 이용한 데이터 증강 처리 시스템 및 그 방법
CN114757307B (zh) * 2022-06-14 2022-09-06 中国电力科学研究院有限公司 一种人工智能自动训练方法、系统、装置及存储介质
CN114757307A (zh) * 2022-06-14 2022-07-15 中国电力科学研究院有限公司 一种人工智能自动训练方法、系统、装置及存储介质
CN116451056B (zh) * 2023-06-13 2023-09-29 支付宝(杭州)信息技术有限公司 端特征洞察方法、装置以及设备
CN116451056A (zh) * 2023-06-13 2023-07-18 支付宝(杭州)信息技术有限公司 端特征洞察方法、装置以及设备
CN117910458A (zh) * 2024-01-19 2024-04-19 上海算法创新研究院 一种基于大语言模型的文本数据清洗系统

Similar Documents

Publication Publication Date Title
US11663409B2 (en) Systems and methods for training machine learning models using active learning
US20190180196A1 (en) Systems and methods for generating and updating machine hybrid deep learning models
US20190179903A1 (en) Systems and methods for multi language automated action response
WO2019113122A1 (fr) Systèmes et procédés d'apprentissage automatique amélioré pour des conversations
US11010555B2 (en) Systems and methods for automated question response
US20200143247A1 (en) Systems and methods for improved automated conversations with intent and action response generation
US20210201144A1 (en) Systems and methods for artificial intelligence enhancements in automated conversations
US9268766B2 (en) Phrase-based data classification system
JP6971853B2 (ja) コミュニケーション及びコンテンツからのコミットメント及びリクエストの自動抽出
US9904669B2 (en) Adaptive learning of actionable statements in natural language conversation
US20190286711A1 (en) Systems and methods for message building for machine learning conversations
US10755195B2 (en) Adaptive, personalized action-aware communication and conversation prioritization
US20200143265A1 (en) Systems and methods for automated conversations with feedback systems, tuning and context driven training
US20200143115A1 (en) Systems and methods for improved automated conversations
US9092802B1 (en) Statistical machine learning and business process models systems and methods
US20170193349A1 (en) Categorizationing and prioritization of managing tasks
US20190286712A1 (en) Systems and methods for phrase selection for machine learning conversations
US20200272791A1 (en) Systems and methods for automated conversations with a transactional assistant
US20190286713A1 (en) Systems and methods for enhanced natural language processing for machine learning conversations
US11106871B2 (en) Systems and methods for configurable messaging response-action engine
US20200201913A1 (en) Systems and methods for improved automated conversations with roi metrics and threshold analysis
US20190122236A1 (en) Systems and methods for message cadence optimization
WO2019191337A1 (fr) Systèmes et procédés pour un traitement amélioré de langage naturel pour des conversations par apprentissage automatique
US20230237276A1 (en) System and Method for Incremental Estimation of Interlocutor Intents and Goals in Turn-Based Electronic Conversational Flow
WO2020139865A1 (fr) Systèmes et procédés pour des conversations automatisées améliorées

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18885845

Country of ref document: EP

Kind code of ref document: A1