CA3166389A1

CA3166389A1 - Machine learning of response selection to structured data input including momentum classification

Info

Publication number: CA3166389A1
Application number: CA3166389A
Authority: CA
Inventors: Joshua Howard Levy; Alexa Breann Eun Taylor; Coke Reed
Original assignee: OJO Labs Inc
Current assignee: Royal Bank of Canada
Priority date: 2019-12-31
Filing date: 2020-12-28
Publication date: 2021-07-08
Also published as: WO2021138227A1; GB202209803D0; GB2606099A; AU2020417756A1

Abstract

A machine learning of response selection to structured data input enables a machine to flexibly and responsively actively engage with a response recipient through a device, such as any electronic device connected to a data network. A system of one or more computers can be configured to respond to structured language messages, such as a textual messages, received from user devices with one or more templates that have been ranked using predetermined criteria including a machine learning classification of a momentum of a structured language message using a machine learning model. The momentum classification is correlated with templates. A ranking engine ranks the selection of candidate response templates in accordance with ranking criteria, including the momentum classifications for the candidate response templates. A highest ranked candidate response template is selected to provide a response to the textual messages received from the user devices.

Description

MACHINE LEARNING OF RESPONSE SELECTION TO
STRUCTURED DATA INPUT INCLUDING MOMENTUM CLASSIFICATION
CROSS-REFERENCE TO RELATED APPLICATION
[001] This application is a continuation-in-part of U.S. Patent Application No. 15/992,851, which is a continuation of U.S. Patent Application No. 15/897,885, which is a continuation-in-part of U.S. Patent Application No. 15/826,151 (referred to herein as the "151 Application"), which are all incorporated by reference in their entireties. This application also claims priority to U.S. Provisional Patent Application No. 62/956,166, which is incorporated by reference in its entirety.
BACKGROUND OF THE DISCLOSURE
Field of the Disclosure

[002] The present disclosure relates to supervised learning processing, and more particularly, to a system, method, and computer program product involving machine learning of response selection to structured data input including momentum.
DESCRIPTION OF THE RELATED ART

[003] Supervised learning is a machine learning process that infers a function for determining an output based on training data, and the function is used to map new input data to an output. A
natural language processor (NLP) represents one embodiment of a supervised learning processor.
In general, a natural language processor (NLP) includes one or more NLP models to generate a prediction about and a response to a human-understandable, natural language (NIL) statement. In at least one embodiment, the NIL statement may be a statement, such as a query or command, and the NLP interprets the statement in order to provide a response.

[004] Humans intuitively decide on how to respond to a conversational statement. For example, if a human is asked by an inquiring individual, "How large is the lot at 123 Pecan?"
The human intuitively knows the context of the statement relates to the area of a parcel of real estate at a particular address. The human responder then provides an appropriate response. If the human knows the answer, the human responds to the inquirer with the answer. So, if the lot is 1/2 acre, strictly the answer to the question is "the lot is 1/2 acre."
However, humans can intuitively enhance the response. For example, the human may know the history and specific details about the inquirer and provide a more insightful response that enhances the answer with information the human could anticipate that the inquirer would want to know or utilize semantics appropriate to the inquirer. For example, if the human knows the inquirer has children and would like a swimming pool, the human in addition to providing the size of the lot, the human may have insights into the inquirer and augment the answer with information with such insights, such as the presence or absence of a pool and state the particular schools nearby. Thus, in response to, "How large is the lot at 123 Pecan?" rather the human may respond, "The lot is rather large at 1/2 acre, has a pool, and the nearby schools are highly rated." Additionally, the human can intuitively gauge the acceptability to the inquirer of the response.

[005] However, machines do not have the benefit of human intuition and cannot determine a proper response in the same way as a human. Furthermore, machine responses are often disadvantageously repetitive, 'mechanical,' and easily distinguishable from a human response.
Additionally, the machine responses are not insightful.
SUMMARY OF THE DISCLOSURE

[006] In at least one embodiment, a system of one or more computers perform operations to respond to messages, such as textual messages, from user devices with one or more templates that have been ranked using predetermined criteria including a machine learning classification of a momentum of the messages. At least one embodiment includes a method of machine learning for selecting a ranked response to a message, such as a textual message, received from a user device. An embodiment of the method, operating in an electronic, machine learning processing system, includes receiving messages from user devices after transmission of one or more ranked response templates to the user devices, providing the messages to a momentum classifier implemented as a machine learning model, assigning momentum classifications to the messages using the machine learning model, correlating the templates communicated to the user devices with corresponding momentum classifications, operating a ranking engine to rank the selection of response templates in accordance with ranking criteria, where the ranking criteria includes momentum classifications for the response templates; and selecting a highest ranked response template to provide a response to the message received from the user devices.
Other

7 PCT/US2020/067095 embodiments include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.
BRIEF DESCRIPTION OF THE DRAWINGS
[007] The present disclosure may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings.
The use of the same reference number throughout the several figures designates a like or similar element.

[008] Figure 1 depicts an exemplary machine learning system that includes a learning, ranked response selection module.

[009] Figure 2 depicts one embodiment of the machine learning system that includes a learning, ranked response selection module of Figure 1.

[0010] Figure 3 depicts an exemplary response ranking and selection process.

[0011] Figures 4 and 5 depict structured data input.

[0012] Figure 6 depicts a response template data structure and response template library.

[0013] Figures 7-8 depict selected response template candidates.

[0014] Figures 9-12 depict exemplary response template ranking parameters and results.

[0015] Figure 13 depicts an exemplary computer system.

[0016] Figure 14 illustrates a response analysis system.

[0017] Figure 15 is a flowchart showing exemplary machine learning operations executed in certain embodiments of the disclosed system.

[0018] Figure 16 is a table showing exemplary text messages and corresponding momentum scores for classification by a machine learning model.

[0019] Figure 17 is a table showing exemplary response criterion and corresponding weighting factors.

[0020] Figure 18 depicts an exemplary computer system.

[0021] Figure 19 depicts embodiments of a computer implemented natural language processing system.
DETAILED DESCRIPTION

[0022] A machine learning of response selection to structured data input enables a machine to flexibly and responsively actively engage with a response recipient through a device, such as any electronic device connected to a data network. In at least one embodiment, the response selection module improves response selection to the structure data input by initially filtering a library of templates to identify candidate templates that best respond to the input. In at least one embodiment, the response selection module ranks the identified candidate templates to provide the response to the device. The response selection module learns by receiving feedback, such as a linked recipient action result signal. The linked recipient action result signal tracks activity of a recipient that is linked to receipt of a particular response. As a particular response becomes linked more frequently with activity that is considered a success, the probability of selecting the template from which the response was derived increases. To all the response selection module to continue learning, new response templates are introduced to the library of templates. In at least one embodiment, the response selection module selects the new response templates in accordance with a predetermined function to allow the response selection module to gauge success of the new response templates.

[0023] In at least one embodiment, the response selection module employs multiple mechanisms to provide ongoing improvement of and learning by the response selection module.
Ranking the response templates allows the response selection module to continually learn which response to select and provide to recipients to maximize successful outcomes.
Additionally, in at least one embodiment, the response selection module accesses additional data sources that can be processed to provide insights that may be helpful in better engaging the recipient and resulting in improved success rates. Furthermore, insights can yield proactive engagement with a past recipient. Additionally, successful insights can be incorporated into response templates for future use with other recipients. The same learning process then allows the response selection module to learn based on such new insights.

[0024] In at least one embodiment, the response selection module uses particular ranking criteria to improve learning by the response selection module of when and which response templates to select for particular recipients. In at least one embodiment, the ranking criteria determines a conversion rate of each candidate template that takes into account multiple factors that influence the ranking process. In at least one embodiment, the conversion rate is defined as recipient activities associated with a template relative to total impressions of the template. In at least one embodiment, the multiple factors that influence the conversion rate include weighting response templates based on (i) activities and measures of closeness that correlate the activity, the recipient, and the provided response and (ii) weighting particular activities where some activities are perceived as more valuable than others. For example, in the context of selling real estate, if the linked recipient action result feedback signal indicates the recipient sends a reply to a response, the response selection module weights the outcome of this activity. If the linked recipient action result feedback signal indicates the recipient schedules a showing of a house, the response selection module weights the outcome of this activity more heavily.
Thus, the response selection module provides technical advantages by employing multiple learning mechanisms to enhance the response selection module to learn, modify responses, and improve performance over time to develop machine intuition.

[0025] In at least one embodiment, the provided responses include textual inputs, such as text messages, that users send to the electronic, machine learning processing system. In at least one embodiment, the text messages are provided to the input of a machine learning engine, which analyzes the text of the messages and assigns momentum scores to the corresponding response templates. In certain embodiments, the momentum scores correspond to an assessment of the satisfaction of the users as the users proceed to a targeted goal of the processing system, such as the purchase of a home. As an example, momentum scores may be used as predictors of the likelihood that users will to continue engagement with the system when certain response templates are used. In at least one embodiment, momentum scores may be used in the selection of the highest ranked, candidate response templates that are used to communicate with the users.

[0026] Figure 1 depicts an exemplary machine learning system 100 that includes a learning, ranked response selection module 101 that provides responses to structured data input. The response selection module 101 can be implemented as a special purpose hardware machine

27 PCT/US2020/067095 including as data processing system configured to execute instructions to implement the response selection module 101. The response selection module 101 receives structured data input, utilizes filtering criteria to select candidate response templates from a library of templates, ranks the candidate response templates in accordance with ranking criteria, and sends a response derived from the highest ranked response to device 118. The ranking criteria is adaptive to feedback, such as the linked recipient action result feedback signal 134 or feedback from a human that monitors the structured data input and selected candidate templates. In at least one embodiment, the ranking criteria adapts to the feedback by modifying parameter values of the ranking criteria that improve template selection and, thus, responses to structured data input.
[0027] In at least one embodiment, the response selection module 101 can receive structured data input from multiple sources and from multiple types of sources. In at least one embodiment, the structured data input has a natural language processing (NLP) output schema to allow the response selection module 101 to seamlessly interact with natural language processor systems, such as NLP system 100. In at least one embodiment, the NLP output schema refers to a schema utilized by a NLP system, such as the schema used in the exemplary response prediction input data 500 (Figure 5). In at least one embodiment, the NLP system 100 interacts with the data processing system 110 and receives a natural language statement submitted by an indirect requestor device(s) 108 as described as the "requestor device(s)" in the US
10,013,654, issued July 3, 2018, and titled "COOPERATIVELY OPERATING A NETWORK OF SUPERVISED
LEARNING PROCESSORS TO CONCURRENTLY DISTRIBUTE SUPERVISED
LEARNING PROCESSOR TRAINING AND PROVIDE PREDICTIVE RESPONSES TO
INPUT DATA" (the '654 Patent), which is incorporated herein by reference. The device(s) 108 is referred to as an "indirect" requestor device in Figure 1 because the response selection module 101 receives structured data input indirectly from the device(s) 108. The data processing system 110 utilizes the NIL statement 106 and contextual data 112 to generate the input data 102 as, for example, described in the '654 Patent. In at least one embodiment, contextual data 112 includes conversation state data and user profile data as described in the '654 Patent.
User profile data represents factual and behavioral information about the user. The NLP system 100 includes natural language (NIL) model(s) 130 that generate the structured data input (SDI) 105 as described in the '654 Patent. In this embodiment, the structured data input 105 is response prediction input data also as described in the '654 Patent.

[0028] In at least one embodiment, the response selection module 101 also receives structured data input directly from the direct requestor device(s) 114. In at least one embodiment, the direct requestor device(s) 114 access the insight/other data 116, which includes information that may provide greater insight into users of the indirect requestor device(s) 108.
For example, if the insight/other data indicates that one of the users has pets and has searched for houses with particular criteria, the direct requestor device 114 could generate structured data input 115 asking if there are houses that meet the criteria that are within a certain distance from a park and indicate that the response selection module 101 send the response to device 118, which in this instance represents the device 108. In at least one embodiment, the response selection module 101 selects a response in accordance with filter criteria derived from the structured data input 115 and informs the user of a house meeting the user's criteria and enhancing the response with the additional information about a nearby park.

[0029] The response selection module 101 can also obtain information from external data source(s) 113 that can include additional information that might be relevant to the user of device 118. In at least one embodiment, the response selection module 101 can utilize the additional information to derive filter criteria to select a response template that may have a higher chance of success in directing the user to a preferred activity.

[0030] The linked recipient action result feedback signal 134 represents data that correlates to an activity of a user of the device 118 that is linked to a response 115 provided to the device 118 by the response selection module 101. In at least one embodiment, an external data source(s) 113 receive the feedback signal 134, and, in at least one embodiment, the response selection module 101 receives and stores the feedback signal 134. The source (not shown) of the feedback signal 134 can be any device that can transmit data related to the user of device 118. For example, the source can be an electronic device of a sales person that has information that the user of device 118 performed an activity related to the response 115 and provides this information. The source can be an application that allows the user to communicate the feedback signal 134 directly, such as a direct reply to machine learning system 100 or other action that correlates the response 115 to an activity of the user within a window of time that allows the response selection module 101 to infer that the user's action was linked to the response 115.

[0031] The feedback signal 134 enables the response selection module 101 to learn and improve performance by correlating particular actions with a response 115. As subsequently described in more detail, when the response selection module 101 correlates the action and responses, in at least one embodiment, the response selection module 101 adjusts template selection ranking criteria accordingly. By adjusting the ranking criteria, the response selection module 101 can improve ranking and selection of response templates.

[0032] Figure 2 depicts an exemplary machine learning system 200 that includes learning, ranked response selection module 202, which represents one embodiment of the response selection module 101. As previously discussed in conjunction with the response selection module 101, the response selection module 202 utilizes the data from data sources 203 to select and rank response templates from response template library 208. The structured data input 204 represents structured data input from any source, such as NLP system 100 and device(s) 114.

[0033] Figure 3 depicts an exemplary response ranking and selection process 300. In at least one embodiment, the machine learning system 200 operates in accordance with the process 300.
Referring to Figures 2 and 3, in operation 302 the response selection module 202 receives the structured data input 204, which is constructed in accordance with a natural language processing output schema.

[0034] Figures 4 and 5 depict exemplary structured data input 400, which is the response prediction input data from the '151 Application. The structured data input 400 includes an incoming message 404 that asks, "How many bedrooms on the first floor?" The contextual data 406 is structured using a natural language processing output schema that includes various parameters such as intents path data 402, i.e. intents path = property details, rooms, bedrooms, room, count. Normalized entities 408 with entity names "listing" and "which,"
and associated entity values of "tx actris 1234567" and "first floor," respectively.

[0035] Referring to Figures 2-5, operation 302 receives the structured data input 204. The structured data input 204 includes a statement to which the response selection module 202 selects one or more responses. The incoming message 404, "How many bedrooms on the first floor?"

represents one example of a statement contained in the structured data input 204. Operation 304 derives template filter criteria utilizing the structured data input 204. The structured data input 204 has a structured natural language processing output schema that allows the response selection module 302 to identify specific parameters within the structured data input 204 from which the response selection module 202 formulates filtering criteria to identify candidate response templates from the response template library 208 that are responsive to the statement.
To effectively function as filtering criteria, the response templates 208 include corresponding parameters and values that can be queried against to identify the candidate response templates.

[0036] Figure 6 depicts an exemplary response template data structure 602 and response template library 604. The response template 602 is a data structure that includes filter parameters 606. The depicted filter parameters 606 are illustrative and can be replaced with any filter parameters that allow the response selection module 202 to identify candidate response templates. The filter parameters 602 include <INTENT PATH>, <ENTITY NAME(S)>, and <ENTITY VALUES> that correspond directly to the same type of parameters in data 402 and 408 in the structured data input 204. The filter parameters 606 can include additional parameters and values that may not correspond to parameters in structured data input 204, such as user profile and group profile data in the contextual data 112, insight parameters in the insight/other data 116, and external data in external data source(s) 113. For example, the structured data input 204 may include specific contextual data about the requestor, such as the requestor's age, preferred commute time range, employment location, pets, children, desired spending range, and desired lot, house size, and number of bathrooms in a home buying context. The external data source(s) 113 can include additional information such as details about a home that can be accessed using a multiple listing service (MLS) number, map distances, school districts, school ratings, etc.

[0037] The insight/other data 116 can overlap with other data sources but can also include additional information that, for example, may be derived from data from other data sources.
Such additional filter parameters are illustratively represented by the <EXTERNAL DATA> and <PROFILE DATA> filter parameters. Accordingly, the filter parameters 602 in the response template can be structured to be responsive to inquiries based on such additional parameters and values. The filter parameters 606 can also include disqualifying data that prevents a response template from being selected. An example disqualifier is if the dynamic content 608 includes mention of an object, such as a pool, and the response should not have such contact because, for example, inclusion of the object in the response 115 could be, for example, misleading.

[0038] The response template 602 also includes content data structure 608. The content represents that actual content that can be provided to a device 118. The content can include static and/or dynamic content fields. Dynamic content refers to content that the response selection module 202 populates with data, and the populated data can change depending on, for example, parameters of the structured data input 204. For example, the {ANSWER} field can represent dynamic content such as insertion of a particular address when responding to a real estate related statement. Each [OBJECT] can refer to, for example, static content, such as introductory or concluding phrases. Additionally, the content can include any type of content, such as text, photo, video, and hyperlinks. The response template 602 also includes an identifier (ID) to uniquely identify each response template.

[0039] The response template library 604 serves as a storage repository for the response templates N+1 number of response templates, where N is an integer. In at least one embodiment the response template library 604 is stored in a database and is accessible using database queries, such as structured query language (SQL) queries.

[0040] Referring to Figures 2-6, in at least one embodiment, the response selection module 202 includes a template filter 210 that performs operation 304 to derive the template filter criteria by formulating a query to the response template library 604 to identify response templates having specific parameters and values contained in the structured data input 204. For example, operation 304 utilizes the intent path data 402 of intents path: "property details, rooms, bedrooms, room, count" and the presence of a "listing" entity name to formulate a filter criteria query of select intent _path templates where intent _path= 'property details, rooms, bedrooms, room, count" and required entity names= "listing." Operation 306 applies the filter criteria to the template library 604 to identify and select response templates that meet the filter criteria.

[0041] Figures 7-8 depict selected candidate response templates having ID's 1-5. Each of the candidate response templates meets the filter criteria for 'property details, rooms, bedrooms, room, count" and required entity names= "listing. In at least one embodiment, the candidate response templates have additional qualifiers such as a status code of "ok,"
which indicates the response templates are available for use and a qualifier that response templates are not for multi-answers. Each candidate response template 1-5 also indicates that specific entity values are not required to select the response template. Each response template includes static and dynamic content. For example, response template id=1, has content of ""The home at {{{messageSulfillment.normalized_entitieslisting.string}}l has {{#messageSulfillmentanswer}}{{{value}}l bedroom{{#is_plural}}s{{/is_plural}}{Umessage.fulfillmentansweril." The bracketed fields enable entry of dynamic content that is populated with values from the data sources 203. For example, from structured data input 400, the content in template 1 is "The home at 61204 Mesa has 3 bedrooms," the content in template 2 is "There are 3 bedrooms in the home at 61204 Mesa." The content can also include dynamic insights if available that are populated from data sources 203, such as "The home at 61204 Mesa has 3 bedrooms and a pool," if an insight into the requestor indicates the requestor was previously interested in properties with a pool.

[0042] A ranking engine 212 performs operation 308 and ranks the candidate response templates using ranking criteria. By ranking the candidate templates, the ranking engine 212 allows the response selection module 202 to select the candidate response template with an estimated highest chance of causing the recipient to engage in an activity considered successful, such as replying to the response or taking certain action, like scheduling a showing of a home for purchase. The particular ranking criteria is a matter of design choice.
Exemplary ranking criteria is set forth below:
for each candidate response template, determining the conversion rate in accordance with equation [1]:
Success Count Conversion Rate = [1]
Attempt Count Success Count =1 SC
(- -User + SCGroup SCAU) a Attempt Count =1 I
(-User + 'Group + 'AU) a SCUser = WaWullconversionsa,u (0)11 SCGroup = WaWG1 Ilconvers1onsa,k(0)11 Icex SCAU = WaWAIlconversionsa,A(0)11 Attempt Count =1 I
(-User + 'Group + 'AU) a 'User = WaWUIliMpreSSiOnSa,u(0)11 'Group = WaWG1 llimpress1onsa,k(0)11 Icex 'AU = WaWAIlimpressionsa,A(0)11 wherein:
wa is an activity weight;
wu is a user weight of a user U associated with the structured data input, wG
is a group weight of a group G that includes the user and is a subset of all users of the supervised learning processing system, and wA is a weight for all users A of the supervised learning processing system;
conversion is a conversion associated with the response template for an activity a by a user the user U, the group G, or all users A; and impression is an impression associated with the response template for an activity a by a user the user U, the group G, or all users A.

[0043] In summary, the foregoing ranking criteria determines a conversion rate that is based on conversions and impressions by a user, a group with similar attributes as the user, and all users.
Successes for the user are more indicative of future success than successes by the group and all users, and successes for the group are more indicative of future success than successes by the all users. Furthermore, different activities are considered more valuable than others as previously described. By determining the weighted conversions and activities relative to the total number of impressions, the ranking engine 212 determines a conversion rate that provides a measure of performance for each candidate response template. The ranking engine 212 revises the number of impressions and from feedback signal 134 adjusts the conversion data.
Furthermore, the weights can be adjusted to further enhance the learning of the ranking engine 212.

[0044] In at least one embodiment, the conversion rate is a function of the success values and the attempt values, i.e. conversion rate =f(success, attempt), and not strictly (success/attempt).
For example, the conversion rate function can incorporate distributions, such as beta distributions parameterized by success and attempt values, of conversion rates and ranking of the candidate response template can be based on a random sampling of the conversion rate values in the distribution. Utilizing this 'distribution' based conversion rate function allows a probability of assigning a higher conversion rate based ranking to candidate response templates that do not have the strictly highest (success/attempt) value. In at least one embodiment, the particular distributions are mathematically shaped to provide a probably frequency of ranking a particular candidate response template with the highest conversion rate. By allowing the ranking engine 212 to distribute the highest conversion rate ranking among candidate response templates, the response selection module 202 learns the effectiveness of different response templates.
Additionally, the response selection module 202 can insert new response templates into the collection of candidate response templates to allow the response selection module 202 to learn about the success of the experimental response templates. When adding new response templates that do not have observed conversion and impression values, the conversion rate function can be modified to ensure any number of highest conversion rate values for each candidate response template by, for example, inserting an override factor that forces a high conversion rate of the new candidate response template. The override factor can be, for example, a weight selected from a distribution of weights or a random number that ensures selection of the new candidate response template at some probabilistic frequency. In at least one embodiment, the override factor is determined by an epsilon-greedy function that forces occasional highest ranking and selection of the new candidate response template.

[0045] Figures 9-12 depict exemplary response template ranking parameters and results. Cells Gl, Ml, and Si are values of respective weights wu, wG, and wA. Each weight from the user to group to all is an order of magnitude larger, which represents the relative importance of the conversion feedback. Cells E2, K2, and Q2 represent weight wa=1 for a user activity of "Got Response," which is receiving a reply to the response derived from the particular response template. Cells G2, M2, and S2 represent weight wa=4 for a user activity of "Showing" of a property because showing is more valuable than receiving a response. The ranking results include "Prior" and "Observed" results. "Prior" is inserted as a value that represents estimated performance to initialize the ranking data so that the conversion rates are more meaningful.
"Prior" and "Observed" values are added together for purposes of values in Equation [1]. The weighted conversions, weighted impressions, and overall conversion rates are computed in accordance with Equation [1].

[0046] The response selection module 202 includes a populating engine 214 that in operation 314 populates any dynamic fields in the candidate response templates and converts the populated content into a response message ready for sending. Population can occur before or after selection of the response template. Operation 312 determines whether the response selection module 202 provides the ranked candidate templates to a human artificial intelligence technician (AIT) 218 to allow the AIT 218 to select the response to send to device 118 or allow the response selector 216 to directly select and send the response to the device 118. The function to determine the outcome of operation 312 is a matter of design choice. In at least one embodiment, the function of operation 312 relies on a confidence level assigned to the response selection module 202 for specific structured data input and responses. If operation 312 selects the AIT 218, the response selection module 202 provides the candidate response templates to the AIT 218 with ranking information for selection. Operation 318 provides a response derived from the selected response template to the device 118. In at least one embodiment, the derived response is the populated content from the selected response template. The selection made by the AIT 218 is fed back to the ranking engine 212 to allow the ranking engine to revise the impression counts for the selected response template and to correlate any feedback signal 134 to the selected response template. If operation 312 does not select the AIT 218 to select the template and send the response, in operation 322 the response selector 216 selects the response template in accordance with selection criteria and provides the response template content to the device 118. The selection criteria is a matter of design choice. In at least one embodiment, the selection criteria selects the template with the highest conversion rate.
However, such criteria can quickly eliminate candidate templates from future selection. So, other selection criteria utilizes a distribution of the conversion rates of the candidate templates to proportionately select candidate templates in accordance with their conversion rates. Additionally, the selection criteria can use an override to select new candidate templates or to force selection of particular candidate templates. Thus, the response selection module 202 is able to continue learning which candidate template is best on a user, group, and all users basis.

[0047] Figure 13 depicts an electronic environment 1300 in which certain embodiments of the disclosed system may operate. In this example, a plurality of users execute client applications 1302, 1304, and 1306 at respective devices 1308, 1310, and 1310. In certain embodiments, the client applications 1302, 1304, and 1306 are the same type of application, such as a real-estate application. In certain embodiments, the client applications 1302, 1304, and 1306 include web browser clients and text messaging applications.

[0048] In at least one embodiment, the client applications receive template communications 1324 from a template selection system 1322 over a network 1314, such as the Internet. In certain embodiments, the template selection system 1322 selects templates that are to be sent to the users in the manner described herein.

[0049] In at least one embodiment, the users employ the client applications to send responses 1326 to a response analysis system 1328. In certain embodiments, the responses 1326 are in the form of messages, such as textual messages, that may be analyzed by an NLP
machine language model of the response analysis system 1328. Users provide responses 1326 under various circumstances. For example, users may provide textual answers to questions that are posed to the user in a template. As another example, users may provide textual inquiries, requests, or comments that, although not explicitly related to a response to a particular template, may be indicative of the momentum associated with a particular template. In certain embodiments, the response analysis system 1322 analyzes the text of responses received from the client applications to assign momentum scores to the templates associated with the responses. In certain embodiments, the responses are associated with the templates in time (e.g., received within a time period after the transmission of the template to the user), in sequence (e.g., received in a particular sequence of responses after transmission of the template to the user), or in subject matter (e.g., pertaining to the same subject matter as the template transmitted to the user) to the template selection system 1322. In certain embodiments, the momentum scores are stored along with the templates in a template database store 1324.

[0050] Figure 14 depicts a response analysis system 1400 that may be implemented in certain embodiments of the disclosed system. In the example shown in Figure 14, responses 1402 are provided to a response API 1404, which provides the text from the responses 1402 to the input of a machine learning model 1406. In certain embodiments, the machine language model 1406 has been trained using training data 1408. The training data 1408 includes text messages that have been labeled to train the machine learning model 1406 to assign momentum classifications to the responses 1402. In certain embodiments, the momentum classifications are numerical momentum scores, which can be used as metrics to calculate conversion rates for the templates.

[0051] In the illustrated embodiment, the machine language model 1406 outputs momentum classifications 1410 for each of the responses 1402. In certain embodiments, the machine language model 1406 also provides a confidence score for each of the responses 1402. The confidence score is generated by the machine language model 1406 and corresponds to the accuracy with which the machine language model 1406 has determined that the corresponding momentum classification is correct. In certain embodiments, a classification aggregator 1412 is employed to aggregate the response classifications associated with each of the templates to generate a momentum score for each template so that the momentum classification assigned to a template takes multiple responses into consideration when determining an overall momentum classification for the template. The classification aggregator 1412 generates the overall momentum classification for the template and correlates the momentum classification with the template for storage in the templates and scores storage 1414. Certain embodiments retrieve the momentum classification for the template to determine the conversion rate that is assigned to the template during template ranking operations.

[0052] Figure 15 depicts a flowchart 1500 showing exemplary operations that may be executed in certain embodiments of the disclosed system. In this example, text responses are received at operation 1502 and assigned a momentum score at operation 1504. In certain embodiments, the momentum score of the text response is correlated with the corresponding template at operation 1506. In certain embodiments, the text responses are correlated with the corresponding templates prior to classifying the momentum scores of the text responses. A
determination is made at operation 1508 as to whether more text responses are to be classified in correlated with the templates corresponding to the text responses. If no further text responses are to be classified, the momentum scores for the text responses for each template are aggregated at operation 1510 to provide a single momentum score for each template. At operation 1512, certain embodiments use the aggregated momentum score for each template in determining the overall template conversion score assigned to the template during template ranking operations.

[0053] Figure 16 is a table 1600 showing exemplary text messages and corresponding momentum scores that may be classified by a machine learning model. In this example, the momentum scores range from 1 through 5, with a score of 5 representing the highest momentum.
In this example, text messages of the type shown at 1602 are classified by the machine learning model with a low momentum score of 1. In certain embodiments, a momentum score of 1 indicates that the user wants to stop using the SERVICE. Text messages of the type shown at 1604 are classified by the machine learning model with a momentum score of 2.
In certain embodiments, a momentum score of 2 indicates that the user is having a problematic experience with the SERVICE. Text messages of the type shown at 1606 are classified by the machine learning model with a momentum score of 3. In certain embodiments, a momentum score of 3 corresponds to a neutral or irrelevant comment relating to the SERVICE. Text messages of the type shown at 1608 are classified by the machine learning model with a momentum score of 4.
In certain embodiments, a momentum score of 4 indicates that the user is having a productive experience with the SERVICE. Text messages of the type shown at 1610 are classified by the machine learning model with a momentum score of 5. In certain embodiments, a momentum score of 5 indicates that the user is moving toward a major milestone or the end of the user's journey with the SERVICE.

[0054] Score components for the templates may be defined as a percentage of users in the values range from 0 (nobody) to 1 (all users) associated with a particular response criterion multiplied by a corresponding weighting factor. In certain embodiments, the score components for the templates are added together to arrive at an overall template conversion score, which may be used, for example, during template ranking operations. As an example, the template conversion score for a system implementing a response analysis system may be determined using the equation:

Template Conversion Score = (C1 *-w + (C2 2) + (C3 W3) + (C, w) where Cis a value between 0 and 1 defined as a percentage of users whose actions meet particular response criterion associated with the template; and wx is the weighting factor that is assigned to the response criterion.

[0055] Figure 17 depicts a table 1700 showing exemplary response criterion and corresponding weighting factors that may be used in certain embodiments of the disclosed system. In this example, the values for the percentage of users G meeting the stated response criterion are shown in the RESPONSE CRITERION column. In certain embodiments, the values for G in the RESPONSE CRITERION will range from 0 through 1. Examples of weighting factor values wx that may be used with the values G of the RESPONSE CRITERION are shown in the WEIGHTING FACTOR. As shown in this example, the momentum score is taken into consideration in row 1702 and corresponds to the value for the percentage of users that respond with the message of momentum 4 or 5 within twelve hours of transmission of the template. In this example, the response criterion has a weighting factor of 0.66, that when multiplied by the value for the percentage of users that respond with the message of momentum 4 or 5 within twelve hours of transmission of the template result in a sub-score SS2. A
total conversion factor for the template may be obtained by adding the values of SS1 through SS9 together.

[0056] Figure 18 is a block diagram illustrating an exemplary network environment in which the machine learning system that includes a learning ranked response selection module response selection module and response analysis system may be practiced. Network 1802 (e.g., a private wide area network (WAN) or the Internet) includes a number of networked computer systems 1804 (1)-(N) that can each implement a specialized machine learning system and the learning, ranked response selection module response selection module. Communication between user requestor devices 1806 (1)-(N) and each specialized machine learning system and the learning, ranked response selection module response selection module 1804 (1)-(N) typically occurs over a network, such as a public switched telephone network or cable network of asynchronous digital subscriber line (ADSL) channels or high-bandwidth trunks, for example, communications channels providing Ti or 0C3 service. Requestor devices 1806 (1)-(N) typically access server computer systems 1804 (1)-(N) through a service provider, such as an internet service provider ("ISP") by executing application specific software, commonly referred to as a browser, on one of requestor devices 1806(1)-(N).

[0057] Requestor device 1806(1)-(N) and/or specialized machine learning systems and the learning, ranked response selection module response selection modules 1804(1)-(N) may include, for example, computer systems of any appropriate design, including a mainframe, a mini-computer, a personal computer system including notebook computers, a wireless, mobile computing device (including personal digital assistants, smart phones, and tablet computers).
These computer systems are typically information handling systems, which are designed to provide computing power to one or more users, either locally or remotely. Such a computer system may also include one or a plurality of input/output ("I/O") devices coupled to the system processor to perform specialized functions. Tangible, non-transitory memories (also referred to as "storage devices") such as hard disks, compact disk ("CD") drives, digital versatile disk ("DVD") drives, and magneto-optical drives may also be provided, either as an integrated or peripheral device. In at least one embodiment, the machine learning system and the learning, ranked response selection module response selection module can be implemented using code stored in a tangible, non-transient computer readable medium and executed by one or more processors. In at least one embodiment, the machine learning system and the learning, ranked response selection module response selection module can be implemented completely in hardware using, for example, logic circuits and other circuits including field programmable gate arrays.

[0058] Embodiments of individual machine learning systems 1804(1)-(N) can be implemented on a computer system such as computer 1900 illustrated in Figure 19. The computer 1900 can be a dedicated computer system or a virtual, emulated system located in, for example, a cloud computing environment. Input user device(s) 1910, such as a keyboard and/or mouse, are coupled to a bi-directional system bus 1918. The input user device(s) 1910 are for introducing user input to the computer system and communicating that user input to processor 1913. The computer system of Figure 19 generally also includes a non-transitory video memory 1914, non-transitory main memory 1915, and non-transitory mass storage 1909, all coupled to bi-directional system bus 1918 along with input user device(s) 1910 and processor 1913. The mass storage 1909 may include both fixed and removable media, such as a hard drive, one or more CDs or DVDs, solid-state memory including flash memory, and other available mass storage technology.
Bus 1918 may contain, for example, 32 of 64 address lines for addressing video memory 1914 or main memory 1915. The system bus 1918 also includes, for example, an n-bit data bus for transferring DATA between and among the components, such as CPU 1913, main memory 1915, video memory 1914 and mass storage 1909, where "n" is, for example, 32 or 64.
Alternatively, multiplex data/address lines may be used instead of separate data and address lines.

[0059] I/O device(s) 1919 may provide connections to peripheral devices, such as a printer, and may also provide a direct connection to remote server computer systems via a telephone link or to the Internet via an ISP. I/O device(s) 1919 may also include a network interface device to provide a direct connection to remote server computer systems via a direct network link to the Internet via a POP (point of presence). Such connection may be made using, for example, wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like. Examples of I/0 devices include modems, sound and video devices, and specialized communication devices such as the aforementioned network interface.

[0060] Computer programs and data are generally stored as instructions and data in a non-transient computer readable medium such as a flash memory, optical memory, magnetic memory, compact disks, digital versatile disks, and any other type of memory.
The computer program is loaded from a memory, such as mass storage 1909, into main memory 1915 for execution. Computer programs may also be in the form of electronic signals modulated in accordance with the computer program and data communication technology when transferred via a network. In at least one embodiment, Java applets or any other technology is used with web pages to allow a user of a web browser to make and submit selections and allow a client computer system to capture the user selection and submit the selection data to a server computer system.

[0061] The processor 1913, in one embodiment, is a microprocessor manufactured by Motorola Inc. of Illinois, Intel Corporation of California, or Advanced Micro Devices of California. However, any other suitable single or multiple microprocessors or microcomputers may be utilized. Main memory 1915 is comprised of dynamic random access memory (DRAM).

Video memory 1914 is a dual-ported video random access memory. One port of the video memory 1914 is coupled to video amplifier 1916. The video amplifier 1916 is used to drive the display 1917. Video amplifier 1916 is well known in the art and may be implemented by any suitable means. This circuitry converts pixel DATA stored in video memory 1914 to a raster signal suitable for use by display 1917. Display 1917 is a type of monitor suitable for displaying graphic images. The computer system described above is for purposes of example only.

[0062] Although embodiments have been described in detail, it should be understood that various changes, substitutions, and alterations can be made hereto without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

WHAT IS CLAIMED IS:

1.
A method of machine learning in the selection of a ranked response to a structured data input having a natural language processing output schema received from a requesting device, the method comprising:
in an electronic, machine learning processing system:
receiving the structured data input, wherein the structured data input includes filtering parameters for conversion into response template filtering criteria;
querying a library of response templates to identify candidate response templates that meet the response template filtering criteria to filter the response templates;
receiving a selection of the candidate response templates that meet the response template filtering criteria and respond to the structured input data, wherein the candidate response templates include static data;
receiving messages from user devices after transmission of one or more response templates to the user devices;
providing the messages to a momentum classifier implemented as a machine learning model;
assigning momentum classifications to the messages using the machine learning model;
correlating the templates communicated to the user devices with corresponding momentum classifications;
operating a ranking engine to rank the selection of response templates in accordance with ranking criteria, wherein the ranking criteria includes momentum classifications for the candidate response templates; and selecting a highest ranked response template to provide a response to a device;
deriving the response to the structured data input from the selected, highest ranked, candidate response template;

providing to the response to a recipient device; and providing feedback to the ranking engine to refine the ranking criteria.

2. The method of claim 1 wherein the requesting device accesses data that provides insight into an individual and formulates the structured data input as a proactive query to obtain a proactive response to provide to the recipient device that is germane to the individual based on the insight.

3. The method of claim 1 wherein the requesting device comprises a natural language processor that processes a natural language statement to generate natural language output data, wherein the structured data input comprises the natural language output data.

4. The method of claim 1 wherein the filtering parameters comprise an intent path, entity names, and entity values derived from a natural language statement and structured in accordance with the natural language processed output schema.

5. The method of claim 1 wherein at least a proper subset of the response templates include one or more dynamic fields, the method further comprising:
populating the one or more dynamic data fields of at least one of the candidate response templates with data obtained from the structured data input.

6. The method of claim 1 further comprising:
accessing profiles of one or more users to obtain augmenting data that augments intent data of the structured data input, wherein each of the profiles about a user includes data representing factual and behavioral information about the user;
and augmenting response templates with the augmenting data.

7. The method of claim 1 wherein the ranking criteria comprises a function of a conversion rate of each of the candidate response templates that distributes the highest conversion rate ranking among candidate response templates to allow the machine to learn about each of the candidate response templates.

8. The method of claim 1 further comprising:
for each candidate response template, determining the conversion rate in accordance with equation [1]:
wherein:
wa is an activity weight;

wu is a user weight of a user U associated with the structured data input, wG
is a group weight of a group G that includes the user and is a subset of all users of the supervised learning processing system, and WA is a weight for all users A of the supervised learning processing system;
conversion is a conversion associated with the response template for an activity a by a user the user U, the group G, or all users A; and impression is an impression associated with the response template for an activity a by a user the user U, the group G, or all users A.

9. An apparatus for machine learning in the selection of a ranked response to a structured data input having a natural language processing output schema received from a requesting device, the apparatus comprising:
one or more data processors;
a memory, coupled to the data processors, having code stored therein to cause the one or more data processors to:
receive the structured data input, wherein the structured data input includes filtering parameters for conversion into response template filtering criteria;
convert the filtering parameters into the response template filtering criteria;
query a library of response templates to identify candidate response templates that meet the response template filtering criteria to filter the response templates;
receive a selection of the candidate response templates that meet the response template filtering criteria and respond to the structured input data, wherein the candidate response templates include static data;
receive messages from user devices after transmission of one or more response templates to the user devices;
provide the messages to a momentum classifier implemented as a machine learning model;

assign momentum classifications to the messages using the machine learning model;
correlate the templates communicated to the user devices with corresponding momentum classifications;
operate a ranking engine to rank the selection of response templates in accordance with ranking criteria, wherein the ranking criteria includes momentum classifications for the candidate response templates; and select a highest ranked response template to provide a response to a device;
derive the response to the structured data input from the selected, highest ranked, candidate response template;
provide to the response to a recipient device; and provide feedback to the ranking engine to refine the ranking criteria.

10. The apparatus of claim 9 wherein the requesting device accesses data that provides insight into an individual and formulates the structured data input as a proactive query to obtain a proactive response to provide to the recipient device that is germane to the individual based on the insight.

11. The apparatus of claim 9 wherein the requesting device comprises a natural language processor that processes a natural language statement to generate natural language output data, wherein the structured data input comprises the natural language output data.

12. The apparatus of claim 9 wherein the filtering parameters comprise an intent path, entity names, and entity values derived from a natural language statement and structured in accordance with the natural language processed output schema.

13. The apparatus of claim 9 wherein at least a proper subset of the response templates include one or more dynamic fields, and the code further causes the one or more data processors to:
populate the one or more dynamic data fields of at least one of the candidate response templates with data obtained from the structured data input.

14. The apparatus of claim 9 wherein the code further causes the one or more data processors to:
access profiles of one or more users to obtain augmenting data that augments intent data of the structured data input, wherein each of the profiles about a user includes data representing factual and behavioral information about the user; and augment response templates with the augmenting data.

15. The apparatus of claim 9 wherein the ranking criteria comprises a function of a conversion rate of each of the candidate response templates that distributes the highest conversion rate ranking among candidate response templates to allow the machine to learn about each of the candidate response templates.

16. The apparatus of claim 9 further comprising:
for each candidate response template, determining the conversion rate in accordance with equation [1]:
wherein:
wa is an activity weight;
wu is a user weight of a user U associated with the structured data input, wG
is a group weight of a group G that includes the user and is a subset of all users of the supervised learning processing system, and WA is a weight for all users A of the supervised learning processing system;
conversion is a conversion associated with the response template for an activity a by a user the user U, the group G, or all users A; and impression is an impression associated with the response template for an activity a by a user the user U, the group G, or all users A.

17. A non-transitory, computer program product comprising code stored therein and executable by one or more processors to cause machine learning in the selection of a ranked response to a structured data input having a natural language processing output schema received from a requesting device, wherein the code is executable to cause the one or more data processors to:
receive the structured data input, wherein the structured data input includes filtering parameters for conversion into response template filtering criteria;

convert the filtering parameters into the response template filtering criteria;
query a library of response templates to identify candidate response templates that meet the response template filtering criteria to filter the response templates;
receive a selection of the candidate response templates that meet the response template filtering criteria and respond to the structured input data, wherein the candidate response templates include static data;
operate a ranking engine to rank the selection of candidate response templates in accordance with ranking criteria;
select a highest ranked candidate response template to provide a response to a device;
derive the response to the structured data input from the selected, highest ranked, candidate response template;
provide to the response to a recipient device; and provide feedback to the ranking engine to refine the ranking criteria.

18. The non-transitory, computer program product of claim 17 wherein the requesting device accesses data that provides insight into an individual and formulates the structured data input as a proactive query to obtain a proactive response to provide to the recipient device that is germane to the individual based on the insight.

19. The non-transitory, computer program product of claim 17 wherein the requesting device comprises a natural language processor that processes a natural language statement to generate natural language output data, wherein the structured data input comprises the natural language output data.

20. The non-transitory, computer program product of claim 17 wherein the filtering parameters comprise an intent path, entity names, and entity values derived from a natural language statement and structured in accordance with the natural language processed output schema.

21. The non-transitory, computer program product of claim 17 wherein at least a proper subset of the response templates include one or more dynamic fields, and the code further causes the one or more data processors to:
populate the one or more dynamic data fields of at least one of the candidate response templates with data obtained from the structured data input.

22. The non-transitory, computer program product of claim 9 wherein the code further causes the one or more data processors to:
access profiles of one or more users to obtain augmenting data that augments intent data of the structured data input, wherein each of the profiles about a user includes data representing factual and behavioral information about the user; and augment response templates with the augmenting data.

23. The non-transitory, computer program product of claim 17 wherein the ranking criteria comprises a function of a conversion rate of each of the candidate response templates that distributes the highest conversion rate ranking among candidate response templates to allow the machine to learn about each of the candidate response templates.

24. The non-transitory, computer program product of claim 17 further comprising:
for each candidate response template, determining the conversion rate in accordance with equation [1]:
wherein:
wa is an activity weight;
wu is a user weight of a user U associated with the structured data input, wG
is a group weight of a group G that includes the user and is a subset of all users of the supervised learning processing system, and WA is a weight for all users A of the supervised learning processing system;
conversion is a conversion associated with the response template for an activity a by a user the user U, the group G, or all users A; and impression is an impression associated with the response template for an activity a by a user the user U, the group G, or all users A.

25. A
method of receiving a response generated by machine learning in the selection of a ranked response to a structured data input having a natural language processing output schema received from a requesting device, the method comprising:
in an electronic, machine learning processing system:
receiving the response generated by:
receiving the structured data input, wherein the structured data input includes filtering parameters for conversion into response template filtering criteria;

converting the filtering parameters into the response template filtering criteria;
querying a library of response templates to identify candidate response templates that meet the response template filtering criteria to filter the response templates;
receiving a selection of the candidate response templates that meet the response template filtering criteria and respond to the structured input data, wherein the candidate response templates include static data;
receiving messages from user devices after transmission of one or more response templates to the user devices;
providing the messages to a momentum classifier implemented as a machine learning model;
assigning momentum classifications to the messages using the machine learning model;
correlating the templates communicated to the user devices with corresponding momentum classifications;
operating a ranking engine to rank the selection of response templates in accordance with ranking criteria, wherein the ranking criteria includes momentum classifications for the candidate response templates; and selecting a highest ranked response template to provide a response to a device;
deriving the response to the structured data input from the selected, highest ranked, candidate response template;
providing to the response to a recipient device; and providing feedback to the ranking engine to refine the ranking criteria.

26. The method of claim 25 wherein the requesting device comprises a natural language processor that processes a natural language statement to generate natural language output data, wherein the structured data input comprises the natural language output data.

27. The method of claim 25 wherein the filtering parameters comprise an intent path, entity names, and entity values derived from a natural language statement and structured in accordance with the natural language processed output schema.

28. The method of claim 25 wherein at least a proper subset of the response templates include one or more dynamic fields, the method further comprising:
populating the one or more dynamic data fields of at least one of the candidate response templates with data obtained from the structured data input.

29. The method of claim 25 wherein the response is further generated by:
accessing profiles of one or more users to obtain augmenting data that augments intent data of the structured data input, wherein each of the profiles about a user includes data representing factual and behavioral information about the user;
and augmenting response templates with the augmenting data.

30. The method of claim 25 wherein the ranking criteria comprises a function of a conversion rate of each of the candidate response templates that distributes the highest conversion rate ranking among candidate response templates to allow the machine to learn about each of the candidate response templates.