JP2020057386A - Directing trajectories through communication decision tree using iterative artificial intelligence - Google Patents

Abstract

To configure artificial-intelligence (AI) decision nodes throughout a communication decision tree.SOLUTION: The decision nodes can dynamically define iteration data corresponding to a trajectory through a decision tree to support successive iteration of AI models.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit and priority of US Provisional Application No. 62 / 566,26, filed September 29, 2017, which is incorporated by reference in US application Ser. No./007677 is a continuation-in-part application. Each of these applications is incorporated herein by reference in its entirety for all purposes.

Background Technological advances have improved the convenience and complexity of many types of communication channels. Also, advances in data storage and networks have increased storage capacity. This allows a greater amount (and type) of data to be stored in the data source and to be transmitted in the future. Thus, a data source can be arranged to deliver many types of data between multiple data channels at any time in the future. When considering delivery of multiple related content rather than a single delivery, the options for delivering the content will explode. Often, content providers indiscriminately provide each data interstar with the same content over the same communication channel by configuring one or more static rules. Although communication specifications for receiving different data requests may be different, rules can be configured to respond indiscriminately and consistently to data requests. While this approach provides simplicity of configuration and deterministic operation, it does not address the potential variability of the overall data ingestor and cannot handle requests optimally.

SUMMARY In some embodiments, a computer-implemented method is provided. Access data structures. The data structure represents a communication decision tree configured to dynamically define individual paths using a machine learning technique. The path through the communication decision tree indicates a series of communication specifications. The communication decision tree includes a plurality of branch nodes. Each branch node of the plurality of branch nodes corresponds to an operation point configured to specify a direction of a predetermined route. At the first time, it is detected that the route passing through the communication decision tree has reached the first branch node among the plurality of branch nodes. A route is associated with a particular user. In response to detecting that the route has reached the first branch node, first learning data generated by processing the first user data using a machine learning technique is obtained. The first user data includes user attributes of a plurality of other users. Further, one or more specific user attributes associated with the specific user are obtained in response to detecting that the route has reached the first branch node. One or more first communication specifications are specified based on the first learning data and one or more specific user attributes, and the first content is provided to a specific user according to the one or more first communication specifications. Sent to the associated user device. At a second time after the first time, it is detected that the path through the communication decision tree has reached the second branch node among the plurality of branch nodes. In response to detecting that the route has reached the second branch node, second learning data generated by processing the second user data using a machine learning technique is obtained. The second user data includes at least some user attributes that are not included in the first user data. Further, in response to detecting that the route has reached the second branch node, one or more second communication specifications based on the second learning data and at least a portion of the one or more specific user attributes. Is identified, and the second content is transmitted to the user device according to one or more second communication specifications.

  In some embodiments, a computer program product tangibly embodied on a non-transitory machine-readable storage medium is provided. The computer program product may include instructions configured to cause one or more data processors to perform some or all of the operations of one or more of the methods disclosed herein.

  In some embodiments, one or more data processors and, when executed on one or more data processors, perform one or more operations of some or all of the one or more methods disclosed herein. A non-transitory computer-readable storage medium including instructions to be executed by the data processor is provided.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

It is a block diagram showing a dialog system. FIG. 4 is a diagram showing a template used to generate a mail communication. FIG. 9 is a diagram illustrating a template used to generate an application notification communication. It is a figure showing an example of a communication decision tree. It is a figure corresponding to a user device and showing an example of the course extended through a communication decision tree. FIG. 4 illustrates an example interface for defining a communication decision tree. FIG. 9 illustrates an exemplary parameter definition interface for a switch icon. FIG. 9 illustrates another exemplary parameter definition interface that includes options for biasing the display of various content in a communication. FIG. 9 illustrates another exemplary parameter definition interface that includes options for biasing the use of various communication channels to send communications. 5 is a flowchart illustrating a process for specifying a direction of a path through a communication decision tree using machine learning, according to some embodiments of the present invention. FIG. 4 is a flowchart illustrating a process for defining a machine learning-based communication decision tree using an interface that supports configurable visual elements. FIG. 4 illustrates an exemplary progressive sparse decision tree, according to some embodiments of the present invention. FIG. 4 illustrates an exemplary representation of data processing that occurs during implementation of a progressive screening decision tree, according to some embodiments of the present invention. 5 is a flowchart illustrating a process of advancing a path through a decision tree using a machine learning model iteratively in accordance with some embodiments of the present invention.

Description In some embodiments, systems and methods are provided that facilitate iteratively identifying communication specifications using machine learning data iteratively. More specifically, a communication decision tree including a plurality of nodes is generated. Each node may correspond to, for example, a detected event or a branch node. The branch node is coupled to a plurality of next nodes that correspond to the determination of the communication specification and indicate communication corresponding to one or more specific communication specifications. Individual paths through the communication decision tree may correspond to individual users and / or one or more specific user devices. Individual paths can extend through a particular subset of nodes. The nodes in the subset may be specific operations initiated by a user and / or specific operations initiated on a particular user device of one or more specific devices, or a particular operation of one or more specific devices. 5 shows the characteristics of the communication sent to the user equipment of the first and / or the determination on the specification of the future communication. For example, future communication specifications may indicate the transmission time, the device being transmitted, the type of communication channel used for transmission, and / or the type of content being transmitted. In some cases, natural language processing may be used to assign one or more categories to each of the one or more content objects sent to the training set and / or to each of the one or more content objects that can be sent. . The communication specification may specify a particular category of the transmitted content.

  Each communication specification can be determined based on current data corresponding to a user and / or a particular user device, a machine learning model, and one or more learning parameters of the machine learning model. The parameter is associated with the plurality of other users and indicates one or more attributes of each user of the plurality of other users and / or indicates a plurality of events (e.g., user initiated actions or characteristics of communications transmitted to the user). Learning can be performed based on the data. Also, the parameters can be learned based on a target path (e.g., specified by a client) corresponding to a particular operation in the communication decision tree and / or a user initiated action.

  The communication decision tree can be configured to include a plurality of branch nodes. As a result, a plurality of communication specifications can be determined for a single route. In some cases, each decision is made using the same type of machine learning algorithm (eg, a supervised regression algorithm). However, a different algorithm can be used for each branch node. This causes the branch nodes to differ, for example, with respect to the type of input processed for each path and / or the learning parameters used to process the input corresponding to the path. In various cases, different branch node algorithms can be trained to optimize the same or different variables (eg, based on the identification of the same or different target nodes). Branch nodes may be different for the type of input processed by the algorithm, and the type of profile data that may be processed for a particular user may be different (eg, by monitoring the interaction). Profile data can be accumulated over time). Further, the learning data associated with any node may change over time (due to continuous and / or iterative learning).

  As an example, upon detecting that the profile data corresponding to the user includes at least a predetermined set of fields of information, the path of the user can be initialized. The profile data may be collected from one or more sessions associated with the user using one or more web servers, or may be obtained from a remote data source. In some cases, the user device automatically detects at least some of the profile data (e.g., automatically configured for communication, unique device identifier, MAC address, browser type, browser version, operating system (Via header information to identify the type, operating system version, device type, device language, etc.) to the web server. In some cases, the communication includes data representing user input (eg, text entered in a web form, link selection, and page navigation) and may be recorded as profile data.

  Initializing the path can include identifying a first node in the communication decision tree. The first node may include a first branch node. The first decision node determines a content object (e.g., identifies various groups of items and / or information associated with the website) transmitted from the plurality of content objects to the user device via email communication. May be supported. The first decision node may correspond to a decision regarding when to send the mail within two days. The determination may be based on first learning data indicating a type of object and / or a communication time most likely to produce a target result for a particular type of user. For example, the goal result may be that a user invokes a page on a website by clicking a link in an email and / or how the user responds to an exchange (eg, purchasing an item shown on the website). May include interacting with a website. The first learning data is that, of the three content objects, the predictive factor for the content object that produces the target result more effectively indicates whether the user uses the mobile device most frequently (compared to a laptop or computer). Or, it can indicate that the user's age and the previous mail operation instruction regarding the type of the content object on the link clicked by the user are included.

  Once the mail has been sent, the path can be extended to the node representing the sent content until the next event is detected. The next event may include, for example, activating a link in the website and may indicate that the user is participating in a current session in the website. Upon detecting such an event, the path is extended to a second decision node to construct a web page on the requested website (eg, whether or not to include dynamic content objects and / or various How to place the content objects). In this example, the second learning data indicates a configuration that is most likely to produce the same target result for a particular user. For example, the second learning data indicates that of the four configurations, the prediction factor for the configuration that more effectively achieves the desired result is whether the user uses the mobile device most frequently (compared to a laptop or computer). Or include the browser type, the current location of the user device, and the current time of the user location. When the web page is transmitted (configured according to the decision made at the second decision node), the path can be extended to a node representing the composition of the transmitted web page. Upon detecting various user-initiated events, system-initiated events, or external events (e.g., a lapse of a predetermined time since a previous event), the path can continue to extend.

  In this example, the target results are similar across multiple decisions. However, instead of identifying a static workflow for the actions to be performed, or instead of determining a user-defined sequence of actions to be performed, the techniques disclosed herein use the current profile data, current learning data, And making decisions regarding individual actions based on the detection of the current event. Machine learning is performed iteratively throughout the life cycle of a particular path to identify the fractional actions to be performed. The method may facilitate achievement of goals by efficiently utilizing the data (eg, because the expansion and / or evolution of learning and / or profile data can be used to determine intermediate processes). In addition, the method may alter the definition and / or constraints of the machine learning technique (e.g., initiated by the client), thereby affecting the path (e.g., where the change has already been initiated). ) A quick effect can be obtained. For example, the client may change the goal from exchanging to maintaining the user device on the website for at least the duration of the session. By changing and immediately taking effect of the machine learning model parameters associated with the various branching nodes, it is possible to influence the paths that were initiated before reaching the nodes thereafter.

  The determination of communication (and / or direction through the communication decision tree) may be based on anonymized data or partially anonymized data. Either or both of the anonymized data or partially anonymized data can be constructed from anonymized data, partially anonymized data or non-anonymized data provided by one or more providers or clients. For example, the remote user data management system can receive partially anonymized or non-anonymized data from one or more data providers, hide or delete fields of individual records according to data privacy rules, and / or Alternatively, the field values of the user sub-population can be tabulated according to data privacy rules. As described herein, anonymized data or partially anonymized data is data that is PII stripped and / or aggregated such that individual data values cannot be associated with a particular person or user beyond a certain probability. Data. Thus, the anonymized or partially anonymized data has sufficient data values to prevent identifying a particular person as a particular user, or at least with certain probability identifying a particular person as a particular user. Don't or hide. For example, anonymized data or partially anonymized data may not include name, email address, IP address, physical address and / or phone number in profile data. The anonymized data or partially anonymized data may include or exclude certain demographic data, for example, age, city, occupation. In some cases, anonymized data or partially anonymized data can be collected and some data can be processed according to privacy rules, ordinances and laws. Anonymized data or partially anonymized data may include IP address ranges, postal codes, dates, previous online inquiry categories, races that may be collected in accordance with various privacy policies and laws that limit the disclosure of personally identifiable information (PII). , Gender, age, purchase history and / or browsing history, etc., may be included.

  In some cases, the anonymized data or partially anonymized data can be used to generate and / or update learning data (eg, one or more parameters) associated with a particular machine learning configuration. Such training can remove these fields from the data record, for example, because they do not need to utilize or use data fields such as contact information. As another example, one or more sub-populations can be generated based on the value of a particular field, and then the particular value of the field can be replaced using the identifier of the sub-population.

  In some cases, the profile data corresponding to the particular user being determined includes anonymized data or partially anonymized data. For example, the system can detect that a route has reached a branch node and can request data from a user data management system (e.g., using a device identifier or other identifier associated with the route). The system may generate profile data including, for example, one or more specific anonymized field values, one or more generalized (eg, assigned to category) field values, and / or deleted field values. Can be returned. When the non-anonymized field value is provided by the client being determined (e.g., by collecting using a data collection feature embedded in the web page and / or by sending from the client), In this case, non-anonymous field data may be included in the profile data when the determined client can be accessed (eg, via a data sharing agreement). The system can also return population data that indicates the relationship between the field values (eg, can be learned and / or evolved over time). Using this population data, the value or category of the missing field can be estimated.

  FIG. 1 shows a block diagram of a dialog system 100. The machine learning data platform 105 includes one or more cloud servers and can be configured to receive user data from one or more client systems 105. The user data may be anonymized user data or partially anonymized user data (stored in anonymized user data store 115) and / or client available secure users (stored in client available secure user data store 120). Can contain data. The secure user data available to the client may or may not be anonymized at a lower level than the anonymized user data. Once received, secure user data available to the client is securely stored in association with the client's identifier. Therefore, other clients cannot call this data. This data can be stored in a multi-tenant cloud storage system. Thus, multiple clients can call one server or multiple servers by logging in to a central location. Access to specific data is controlled according to clients authenticated to the cloud storage system. Anonymized user data or partially anonymized user data may be individually configured for various clients (eg, depending on data provided by the client and / or data sharing agreements associated with the client), It does not need to be configured. Accordingly, the profile data populator 122 of the machine learning data platform 105 can generate profile data corresponding to one or more individual users for a particular client, and generate profile data for an individual client. You can customize the field values included in.

  In some cases, the profile data populator 122 can enhance the profile data set by using the partially anonymized user data to supplement the client available user data. This allows the client-specific learning data (stored in client-specific learning data store 130) to be defined (or aggregated) for a particular user. For example, profile data of data available to the client can be mapped to one or more datasets of anonymized user data or partially anonymized user data. This allows a richer data set to be used for machine learning analysis. Mapping can be performed using overlapping data (eg, IP address, purchase time, pseudo-random user identifier specified by the client included in anonymized user data or partially anonymized user data).

  The machine learning model configurator 123 may, for example, determine based on specific goal results, available training data, one or more client-specific constraints, and / or possible actions indicated by the communication decision tree and / or by the client. A predetermined machine learning model can be configured. Configuring a machine learning model includes defining one or more parameters of a particular instance of the model (eg, an instance associated with a particular branch node, client, and / or period).

  Each parameter may indicate a relationship and / or correlation between user attributes (stored in learning parameter data store 125). The parameter may include a weight indicating the method and / or degree by which the first user attribute estimates the second user attribute or a weight corresponding to an indication as to whether and / or the degree of the target result has occurred. The weights may be defined along a discrete or continuous value range and / or may be binary.

  As an example, the parameter can indicate an attribute of the attribute set from which a future occurrence of a particular type of exchange event can be inferred. For example, if a user accesses a web page related to the “Travel” tag three or more times in the last month, it may be inferred that a luggage will be purchased. As another example, visiting a movie review web page on a predetermined date may be presumed to purchase an online rental of the movie later. One can learn to identify indirect relevance and trends, for example, the inverse correlation between the age of the user and the average time online each day. Each parameter may be associated with the strength and / or confidence of the relationship, and may optionally be associated with a direct association between the collected data points and the conclusions made. Each direct association includes the probability that the data at the start of the association is accurate and the other probabilities that the association itself is accurate.

  Although not required, client-specific parameters can be generated by performing configuration using profile data available to the client. The client-specific parameters are, for example, modified parameters generated using anonymized profile data or partial anonymized profile data.

  Learning data can be generated using various machine learning techniques. For example, machine learning techniques include decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, expression learning, similarity and metric learning, sparse dictionary learning , Genetic algorithms, or rule-based machine learning. In some cases, machine learning techniques include ensemble techniques. Ensemble techniques learn the weights between ensembles and apply them to results generated from various underlying techniques (eg, two or more techniques described above). The weight between ensembles may be specified, for example, based on accuracy, speed and / or resource usage associated with the underlying technology.

  The training of machine learning techniques (to identify one or more parameters) may include an observed input set (eg, marketing email content, promotion content and / or website configuration) and a corresponding output set. (E.g., as a result of a corresponding marketing email, a corresponding promotion, and / or a corresponding website configuration, e.g., the presence or absence of a particular exchange event). By using these observations to identify the relationships and / or trends being modeled, factual information candidates that cannot be obtained based on factual information (eg, based on the next estimated or specific input received (Estimated output) can be achieved. Each estimate includes a confidence or probability, and the set of estimates includes a combination of the confidence or probability.

  Accordingly, the machine learning model configurator 123 can specify model parameters for a particular client system 110 based on, for example, target results, client-specific profile data, and / or machine learning techniques. If desired, the client-specific learning data may be shared with the client system that provided the underlying client-usable profile data. The client system 110 can include a system for hosting one or more websites and one or more apps and / or sending email. For example, the client system 110 may include a web server 135 for receiving and responding to HTTP requests for pages on one or more domains, and a mail server 140 for delivering mail to a user's mail address. . Client system 110 may additionally or alternatively include an app server 145 for receiving requests from apps running on user devices and responding to the received requests. Accordingly, one or more servers on client system 110 detect a request from one or more user devices 150-1, 150-2 and / or one or more user devices 150-1, 150-2. Can be configured to send the content to The user devices 150-1 and 150-2 may include, for example, a computer, a smartphone, a tablet, and the like. In various situations, a single user device may be associated with a single user or multiple users. Further, a single user can be associated with a single user device or multiple user devices.

  The web server 135 and / or the application server 145 may store an indication indicating a request for content (eg, a web page or an app page) from the content library 153 in the client management user data store 150 as user data. it can. The stored data may include automatically detected information (eg, request time) along with information included in the request (eg, device identifier, IP address, requested web page, user input, etc.). . Saving the data can include updating the profile to include the data. Further, web server 135, mail server 140 and / or app server 145 may be delivered to a particular user device (e.g., by specifying a transmission time, a user device identifier, a content-object identifier, and / or a communication type). The data indicating the content can be stored in the client management user data store 150.

  The client system 110 can transmit at least some user data from the client management user data store 150 to the machine learning data platform 105, which sends the data to a client available secure user data store 120. Can be saved. Such data may be sent periodically at regular intervals while requesting client-specific learning data. In some cases, the client system 110 at least partially removes some or all of the user data (e.g., by omitting or hiding the value of at least some fields) before sending the user data to the machine learning data platform. Be anonymous. Therefore, these data are stored on the platform as anonymized user data or partially anonymized user data. In some cases, the data is not at least partially anonymized, but is stored in a client available secure user data store 120 or at least partially anonymized in a machine learning data platform 105. For some datasets, the anonymized or partially anonymized data is received from a third party and stored on the client system 110 after removing the PII so that the non-anonymized data cannot be accessed. In some embodiments, the anonymized or partially anonymized data is natively or partially anonymized. In these embodiments, the website executes a script embedded on the website. These scripts, when executed, collect anonymized or partially anonymized data about a user's access to a website. The script collects only information that is stored in the data cloud and can be collected without knowing the user's personal information. As a result, the user ID cannot be estimated beyond a certain probability.

  The client system 110 can store the machine learning data in the machine learning data store 155. In some cases, the machine learning data may include an indication of one or more decisions made at a branch node of a given path, a communication decision tree, and / or one or more content specifications identified using one or more parameters. including. Machine learning data can be requested and received from machine learning platform 105 and / or can be derived from data from machine learning platform 105. For example, in some cases, the machine learning model configurator 123 may send parameters generated for the client and / or applicable to the client to the client system 110. As another example, the machine learning model implementer 157 may identify one or more specific communication specifications by applying a machine learning model configured with specific parameters to specific profile data, Can be defined for the client (and / or the next node in the path) corresponding to The machine learning model implementer 157 may send an indication indicating the identified communication activity and / or the next node in relation to the path, the user and / or the user device identifier.

  Identifying a next node and / or communication specification may include executing a machine learning model (associated with the current branch node) using the particular profile data and one or more learning parameters. it can. The result may, for example, indicate a characteristic that results in a particular target result (eg, a desired exchange) with a high probability (eg, above a threshold) or the highest probability among various content display characteristics. In some cases, the analysis includes identifying one or more content display characteristics that will most likely result in a particular exchange. In some cases, the analysis balances the probability of producing a particular exchange result with a predetermined cost metric associated with various content display characteristics.

  In some cases, executing a machine learning model with parameters (eg, at machine learning data platform 105 or client system 110) may include performing one or more regression analyzes, eg, using profile data and parameters. Generating a numerical value that can be compared to a threshold of. One or more thresholds can define two or more ranges (eg, open ranges or closed ranges), each range corresponding to a particular next node and / or communication operation. In some cases, executing the machine learning model with the parameters processes at least some of the profile data and at least some of the parameters (e.g., via calculating a difference or calculating a cost using a cost function). This can include generating a result that can be compared to each variable of a set of reference data variables (eg, single values, vectors, matrices, time series). Each variable is associated with a particular next node and / or communication operation and is defined based at least in part on the parameters. The node or communication associated with the reference data variable that showed the closest correspondence by comparison is selected.

  The dynamic content generator 147 can trigger the presentation of the content object according to the selected communication specification. To generate the appropriate instructions, the dynamic content generator 147 first sends the communication channel used to send the object, the type of object sent, the version of the content sent, and / or the object. Identify the time to do it. The identification may be based on, for example, a result of the execution of the machine learning model, a configuration of the machine learning model (eg, which may limit one or more selections), and / or one or more parameters. Good.

  The dynamic content generator 147 can, for example, identify the type of communication to be sent (eg, email, SMS message, browser pop-up window, or push app alert) and send the web server 135 to send the communication. , The mail server 140 and / or the application server 145. The identification can be done explicitly (eg, based on machine learning results, parameters, and / or the configuration of the machine learning model) or implicit (eg, by the selected content object having a particular type). Can be done

  Identifying a Content Object may include selecting a Content Object from existing Content Objects, or may include creating a new Content Object. Content objects can include, for example, web pages, objects in web pages, images, text messages, mail, mail and / or objects in text. In some cases, the results obtained by executing the machine learning model constructed using the profile data identify a particular content object. In some cases, the results identify characteristics of the content (e.g., having a particular metadata category) and / or identify a particular technique for selecting the content. For example, the results may indicate that "tools" are added to the content object as features and / or that the communication includes four content objects corresponding to four (unspecified) different categories. In this case, the dynamic content generator 147 may select a content object from a plurality of possible content objects, for example, using a selection technique indicated by machine learning execution results, parameters, and / or predetermined settings. it can. For example, the selection technique may be a pseudo-random selection technique, a technique for identifying recently added objects, the most possible from multiple possible content objects (eg, having one or more attributes indicated in machine learning results). Techniques for identifying exchange objects can be included.

  In some cases, the time to send the communication is explicitly specified (eg, based on machine learning results, parameters, and / or configuration of the machine learning model). For example, a time range can be defined that starts at the current time and ends at the maximum time specified by the client. The model can evaluate a plurality of possible transmission times regularly divided into time ranges. In some cases, each possible transmission time is considered multiple times in combination with other possible specifications, such as content categories or communication channels. The result of the machine learning model can identify the transmission time with the highest probability of producing the target result. In particular, when considering a combination of specifications, the transmission time may include a combination of times associated with the highest probability. Optionally, by receiving a next request for content (e.g., corresponding to a given website or app) from a user device associated with the result of the machine learning, or according to a predefined transmission schedule. Send the communication immediately.

  In some cases, identifying each specification corresponding to a communication when or before sending the communication (during a task and / or using machine learning models, machine learning configurations, parameters, client rules, etc.). . Thus, all or part of the client control configuration of the communication and / or its transmission may be performed before transmitting the communication. Optionally, after transmitting the communication, at least one specification corresponding to the communication is identified (during a task and / or using a machine learning model, machine learning configuration, parameters, client rules, etc.). Thus, all or a portion of the client control configuration of the communication and / or its transmission may be performed after transmitting the communication. Thus, this post-transmission setting may be based on learning data and / or profile data that was not available before the transmission of the communication. For example, additional profile data corresponding to the user will be available between the first time the mail was sent and the second time the mail was opened. The sent mail can include a script that runs when the mail is opened. The script can cause a request to be issued to specify device properties such as layout and / or application type. The script can pass these properties to the server with a request to indicate the selection of particular content to display the content. Thus, the server may select content and / or identify one or more display configurations based on the particular display information, current profile data and / or current parameters.

  As an additional or alternative example, the communication may include one or more references or links to pages that display the content when opened (eg, in a web browser). The linked page may include content determined by the machine learning engine before or when generating the communication. The page detects a quest to request display of the page (e.g., when a script detects execution of a link) and / or (e.g., as indicated by execution of the script as part of loading the page). It can be configured to include content that is selected or generated when generating or displaying the page. In some cases, the script in the mail identifies the content structure when the mail is being displayed or when the display of the mail is requested. In some cases, a script running on the linked page specifies the content structure.

  As an example, a client system may provide an online purchase of a food product for delivery. The client system can detect that a particular user has seen a menu for a particular restaurant at 2 PM. The client system can acquire a user attribute from the user profile data as client management user data. The client-specific learning data indicates that the user makes a purchase from a restaurant when a mail containing a discount code is sent to the user in the evening (e.g., compared to a low likelihood associated with other types of communication and other times). It can be shown that the probability is 76%. In response to determining that the 76% likelihood exceeds the 65% threshold for sending a discount, mail server 140 sends an email to the user device. This email contains a script that, when executed, will display the restaurant and discount. The user opens the mail at 10 am the next day. When the code is executed, request a restaurant and discount from the client system. Therefore, the client system has received the latest public learning correlation data. The client system inputs the time, the user's location (if currently working), and past purchase information into a decision tree constructed based on the learning data. The client system may have a discount of 10% (eg, to maintain an exchange possibility threshold) and the restaurant may be a side dish close to the user's workplace (eg, to maximize the exchange possibility). Decide that. On the other hand, if the user opens the email the night before, due to different user attributes and learning data (eg, to maximize the potential) from an Indian restaurant near the user's house (eg, A 15% discount (to maintain the gender threshold) is obtained. The email contains a link to order from the side dish shop. When the user clicks the link, the web server determines the content to be displayed, specifically the recommended food. Based on the latest public learning correlation data, salads and sandwiches should be recommended over soups and meat dishes because previous choices have recently become popular (presumed to be popular due to warmer weather) . Thus, this example illustrates dynamically customizing content for a particular user based on the latest learning data and user attributes.

  For example, machine learning data platform 105 may generate up-to-date client data based on communications received from a user device (eg, in response to a workflow operation). For example, the latest client data may include one or more new fields generated based on the header or payload data of the received communication, an indication of whether a particular event was detected (and a detection time), and / or It can include the current or final phase of the workflow that assigned the profile. The machine learning data platform 105 may provide the latest client data to the client system 110 (eg, with a corresponding profile identifier), and the client system 110 may store the latest data in a client-specific learning data store 165. Can be. Although not required, the client system 110 stores the latest data separately from the base profile.

  In some cases, some or all of the machine learning data platform may be incorporated into client system 110. In some cases, client system 110 communicates with the machine learning data platform during the communication decision tree iteration. For example, the client system 110 (eg, the web server 135 or the application server 145 of the client system 110) sends a flag (eg, included in a URL) from a request to request web or application content received from the user device. Can be detected. This flag indicates the relevance to the machine learning-based workflow. The client system 110 can then update the route appropriately by notifying the request to the machine learning model implementer 157.

  The machine learning data platform, client system 110, and user devices 150-1, 150-2 may communicate via a network 160, which may include, for example, the Internet, local area networks, and wide area networks. Various alternatives to the illustrated and described embodiments are also contemplated. For example, the client system 110 can perform some or all of the machine learning. The client system 110 can periodically receive anonymized user data or partially anonymized user data and process it using machine learning techniques.

  Another technique for using and configuring a communication decision tree was filed on June 13, 2018 (see "Methods and Methods for Configuring Communication Decision Trees Based on Connected Positionable Elements on Canvas"). U.S. Application No. 16/007762 (entitled "System") and filed on June 13, 2018 (entitled "De-obfuscation of Data Based on Machine Learning for Data Enrichment"). It is described in detail in U.S. Application No. 16/007787. Each of these applications is incorporated herein by reference in its entirety for all purposes.

  2 and 3 show interfaces 200 and 300 for configuring templates 202 and 302 for communication that are partially configured when detecting or displaying a display process. The configuring includes executing the configured machine learning model using the current learning configuration of the model and the current profile data. The template 202 shown in FIG. 2 includes the template used to generate the email communication, and the template 302 shown in FIG. 3 includes the template used to generate the application notification communication.

  Template 202 includes static text (eg, text 205) and interaction features (eg, buttons 210). Template 202 further represents a particular layout in which three items are arranged linearly above text 205. Further, the template 202 includes a dynamic component (for example, the dynamic text 215 and the dynamic image 220) specified when requesting display of the mail or displaying the mail. Thus, when sending an email communication, the static component locally identifies at least a portion of the current profile data (when detecting a request to view the email) and Request, request identification of a dynamic component, and receive or receive a dynamic component (e.g., identified using current profile data, current anonymized data or partially anonymized data and / or current learning parameters). It may be obtained and / or sent with code configured to generate a complete email based on the template and the dynamic component. Thereafter, the generated mail is displayed.

  Template 302 includes a static layout and a plurality of dynamic text components (eg, dynamic title 310). The template 302 may be configured to be sent with a script to facilitate dynamically identifying each dynamic text component. Upon detecting a request to display a notification (e.g., opening an app, clicking on a notification app element), the script may, for example, identify at least a portion of the current profile data locally, Requesting at least a portion of the profile data, requesting identification of the dynamic text component (e.g., using current profile data, current anonymized data or partially anonymized data and / or current learning parameters). Received or obtained dynamic text component and / or generate a complete notification based on the template and the dynamic text component. Then, the generated mail can be displayed. The interface 300 shows an example of a dynamically generated notification 315 that includes a static layout and certain dynamic text.

  FIG. 4 shows a communication decision tree 400. The communication decision tree 400 includes a departure node 405 where each route starts. In this example, the user has completed two specific actions (eg, initialized two website sessions, purchased two items from the website, and at least two web pages on the website). Navigation), a specific route can be initialized.

  Communication decision tree 400 includes three branch nodes 410, 415 and 420. Each branch node branches to connect to three nodes representing three different actions. The route can be automatically and immediately extended from the departure node 405 to the first branch node 410. This triggers the first decision. Specifically, the first decision involves identifying a communication channel used to send alerts for website features. The alert can include, for example, a static header that is automatically displayed and indicates that a product or discount associated with the website is (usually) available. When the alert detects a request to open a notification, the alert identifies the dynamic at the second branch node 415 (eg, to specifically specify one or more products and / or discounts). It may be further associated with the content.

  The first branch node 410 includes a first operating node 425a representing a mail communication channel, a second operating node 425b representing an SMS message communication channel, and an app-based communication channel (in this case, the notification is installed on the user device). (To be pushed to or pushed by the assigned application).

  The first decision can be made using a machine learning model constructed based on one or more first parameters. The one or more first parameters may be determined dynamically based on anonymized and / or partially anonymized user data and / or client-specific data. For example, anonymized and / or partially anonymized user data was transmitted over each of three different communication channels for different user sub-populations (defined based on one or more user attributes). The alert may indicate the effect of initiating a corresponding website session (e.g., determined based on a tracking link in the alert) and triggering the user to complete the transaction during the session. The anonymized user data and / or the partially anonymized user data may correspond to many different websites and / or websites having one or more particular characteristics. Client-specific data can include data tracking a particular website of interest to a particular client, and can include data that specifically identifies each user that sent various alerts and results. Thus, while the client-specific data is in some respects richer than the anonymized and / or partially anonymized data, the number of users represented by the client-specific data is different from the anonymized and / or partially anonymized data. May be less than the number of users represented by the digitized data. Further, client-specific data may not have associated attribute combinations. For example, if certain clients have not used app-based alerts before, the accuracy of machine learning models that estimate the possible impact of such alerts may be reduced.

  The machine learning model (configured with the first parameter) can use the profile data associated with the route to determine a communication channel to the user. The profile data may include profile data collected by the client (e.g., using metadata, cookies, and / or inputs associated with a previous HTML request from a user device associated with the route). The profile data may further include other profile data requested and received, for example, from a remote user profile data store for collecting and managing profile data from multiple web hosts and clients.

  Once the communication channel has been identified, the path extends to the corresponding operating node (425a, 425b or 425c). Next, an alert is transmitted using the corresponding communication channel. The alert can be configured to automatically identify restricted content and extend the path to the second branch node 410 upon detecting a request to open the alert. Next, a decision can be made at the second branch node 410 to determine the specific content to be displayed in the body of the alert.

  Thus, the second branch node 415 includes a first notification content node 430a representing content identifying a product that the user last viewed on the website, the four notification content nodes (whole user) who viewed on the website last week. It is linked to a second notification content node 430b representing the content specifying the product and a third notification content node 430c representing the content including the designation of the discount. The second decision can be made using a machine learning model constructed based on one or more second parameters. Thus, in some (but not all) examples, the particular configuration (e.g., indicating the weight given to various user attributes considered in all and / or some goal results) may be different. The general machine learning model used to make decisions at the various branch nodes is the same.

  The one or more second parameters can be dynamically determined based on the anonymized user data and / or the partially anonymized user data and / or the client-specific data. However, each of the anonymized user data and / or the partially anonymized user data and / or the client-specific data may be changed after making the first decision. This leads to a difference between the first parameter and the second parameter. Further, the possible operations performed at the second branch node 415 are different from the possible operations performed at the first branch node 410. Therefore, the first configuration and the second configuration may be different. Further, the profile data processed between the first branch node and the second branch node may be different. For example, a client-related application may have been installed on a user device between a process performed on a first branch node and a process performed on a second branch node. Therefore, the application-based notification is notified, for example, at the second branch node, but not at the first branch node.

  Once the content is identified, the route extends to the corresponding content node (430a, 430b or 430c). The corresponding content is transmitted to the user device and displayed on the user device.

  The content can include one or more tracking links to web pages on the website. Upon detecting that the tracking link has been clicked, the path can be extended to a third branch node 420. Thereafter, a decision can be made at a third branch node 415 to determine the particular content to be displayed on the requested web page.

  Thus, the third branch node 420 includes a first web page content node 435a that represents content for identifying four representative products associated with different categories, and four representative products associated with the same category. A second web page content node 435b representing content for identifying a product, and a second representing content for identifying a single product that is presumed to be of interest to a given user based on previous web page interaction data. 3 web page content node 435c.

  The third decision can be made using a machine learning model constructed based on one or more third parameters. The third parameter differs from the first and / or the second parameter by temporary changes to the anonymized user data, partially anonymized user data and / or client-specific data and / or differences in possible operations. There are cases. Further, the profile data processed at the third branch node 420 may include (e.g., detecting new metadata communicated from the user equipment and / or receiving new information corresponding to the profile from the remote system). Profile data processed by the first branch node 410 and / or the second branch node 415 may be different.

  Once the content is identified, the path extends to the corresponding content node (435a, 435b or 435c). The corresponding content is transmitted to the user device and displayed on a corresponding web page on the user device.

  In the communication decision tree 400 shown in FIG. 4, a single decision is made at each communication stage (when sending the notification, when displaying the body of the notification, and when displaying the web page), Alternatively, multiple decisions can be made using a machine learning model. For example, at branch node 410, the communication channel to use and the time to send the notification can be determined (e.g., by specifying a time from a period of time or from multiple possible times). As another example, the transmission time can be individually determined before or after determining the communication channel. Thus, a machine learning model can be configured to generate multiple outputs, and multiple machine learning models can have multiple configurations (each machine learning model has different parameters and / or hyperparameters). Corresponding, individually trained, and / or produce different types of output).

  FIG. 5 shows an example of a path 500 corresponding to the user device and extending through the communication decision tree 400. In this case, the result of the machine learning performed at the first branch node 410 indicates that the mail communication channel is used to send the notification. Thus, path 500 extends to first operating node 425a. Thereafter, an e-mail notification is transmitted to the user device. Since a request for mail content indicating that the user is about to view the mail has been detected, path 500 extends to second branch node 415 and decides to include the content containing the discount designation in the mail. Thus, the path 500 reaches the third notification content node 430c and transmits the corresponding content to the user device.

  Then, when a request for a web page corresponding to the target link in the email is detected, the path 500 reaches the third branch node 420. Machine learning results are generated that indicate that content identifying the four representative products, each associated with a different category, is to be placed on the web page. Thus, the path 500 reaches the first mail content node 435a and sends the corresponding web page content to the user device.

  In the illustrated example, the determination of the first branch node, the second branch node, and the third branch node is made at 5:00 pm on the first day, at 12:00 pm on the second day, and at 6:00 pm on the second day. As shown. The corresponding action is performed immediately. Note that the operation time may be determined according to execution of a machine learning model, a client rule, or another method.

  Further, identifying machine learning based decisions can include implementing one or more additional constraints and / or factors. Alternatively or additionally, the machine learning based decision may be modified based on one or more additional constraints and / or factors. For example, U.S. application Ser. No. 14 / 798,293, filed Jul. 13, 2015 (incorporated herein by reference in its entirety for all purposes) describes a machine learning technique disclosed herein. Additional techniques for dynamically identifying communication characteristics that can be combined are described in further detail.

  In some embodiments, systems and methods are provided that utilize a canvas to facilitate configuring a series of machine learning implementations to partially define a communication exchange. More specifically, it utilizes a canvas on which individual switching visual elements and a corresponding plurality of communicating visual elements can be arranged and connected. Next, a communication decision tree can be generated based on the arranged and connected visual elements. The canvas can be configured to identify one or more communication specifications associated with each communication visual element. Each switching visual element is used to select a particular communicating visual element from a plurality of communicating visual elements coupled to the switching visual element, and may represent a machine learning technique associated with particular parameters learned through training. it can. The canvas can be configured to identify target outcomes (eg, representing user initiated events or communications) that can indicate machine learning choices (eg, for each or all visual elements). Thus, a particular communication visual element selected using machine learning techniques corresponds to a communication specification that is relatively likely to produce a target result (which can be represented, for example, as an event visual element of a communication decision tree).

  A machine learning model can be defined for each switching visual element. Machine learning models can be trained using previous paths for other communication decision trees (however, correspond to communication specifications similar or similar to the communication specifications represented by the communication visual elements in the model being trained) Utilize other communication decision trees that have a communication visual element to perform). For example, for routes that have been routed to trigger communications having particular communication specifications, subsequent user-initiated events represented by those routes (and portions of the routes that represent the occurrence of client-specified target results). ) Can train the model. Additionally or alternatively, when paths associated with the generated communication decision tree appear, these paths can be used to train the model.

  In some cases, the model can be trained with a data set that reflects previous events (eg, paths through the same or different communication decision trees and / or other indications of event sequences) and uses new data Can be extended. The new data may have recently become available (eg, due to newly received form inputs or metadata detection) and may correspond to variable types that change statically or estimable. For example, if the user's age is specified at time X, the user's age at time X−3 years can be calculated. Becomes lower. Training can determine whether various attributes represented in the new data can predict the occurrence of a particular event.

  The interface can be configured to accept an indication indicating a bias applied to various stages of machine learning. For example, for a particular switching element that is coupled to a plurality of particular communicating visual elements, the client may interact with a control slider visually associated with the first visual element to direct the first visual element. Alternatively, it may be indicated to select a route at or away from the first visual element. By setting the metadata to be given to the machine learning model based on the interaction, the corresponding bias can be influenced. In some cases, the metadata corresponds to unlearned hyperparameters used to adjust or constrain learning parameters (such as weights). In some cases, the metadata can be used to define post-processing adjustments that are applied to the results generated by the machine learning model. In some cases, the client or system biases certain communication visual elements if the training data corresponding to the communication specifications represented by the elements (eg, associated with all and / or certain communication phases) is relatively low. .

  In some cases, the client may use the interface to define the structure of the communication decision tree and / or define one or more hyperparameters of a machine learning model that is executed at each decision node. Note that the machine learning model can be defined based on one or more hyperparameters and one or more parameters. Each of the one or more hyperparameters includes a variable that has not been learned by training the machine learning model, and the one or more parameters includes one or more variables that have been learned by training the machine learning model. Thus, for example, the client may determine whether the number of branch nodes observed during execution of an individual machine learning model, the action corresponding to each branch connected to each branch node, the connection between other nodes, or one or more The interface can be configured so that hyper-parameters indicating constraints can be specified.

  FIG. 6 shows an exemplary interface 600 for defining a communication decision tree. Specifically, the interface includes a canvas 605 on which representations of various nodes can be arranged and connected. Interface 600 may include a plurality of icons located on canvas 605 and selected to represent a particular sequence of actions. The plurality of icons may include a start icon 610 representing a start of a communication decision tree. The start icon 610 is associated with configuration logic that, when satisfied, can receive a definition of a condition indicating that a path through the communication decision tree is to be initiated.

  The plurality of icons may further include an end icon 615. A communication decision tree can be defined to indicate that a predetermined route is completed upon reaching the end icon 615. Next, the client represents the various operations and evaluations performed during the route observation by linking the action definition icon and / or the event detection icon located between the start icon 610 and the end icon 615. Can be.

  The operation definition icon included in the plurality of icons may be the switching icon 620. The switch icon 620 corresponds to a branch node for selecting or “switching” a branch. A branch can be selected using the configured machine learning model and profile data. In many cases, the switch icon 620 is linked to multiple possible paths. Possible paths may intersect with another icon (eg, a communication icon, an event detection icon, another switching icon, and / or an end icon).

  As an example of the communication icon, a mail icon 625 indicating that a mail is transmitted to the user device, a text message icon 630 indicating that a text or SMS message is transmitted to the user device, and an application installed in the user device. An application message icon 635 indicating that an alert is to be displayed. In some cases, a possible route indicates that no action is taken (by not providing a communication icon). In the illustrated canvas, the arranged switching icons are linked to three paths: two mail paths (eg, associated with different content and / or transmission times) and one inactive path.

  The event detection icons included in the plurality of icons may include a target detection icon 637 indicating that an event corresponding to one or more target results of the machine learning technique has been detected. The target detection icon 637 and / or another event detection icon may indicate, for example, that a notification has been opened, that a target link included in the notification has been executed, that a user device associated with the route has initiated a website session. May indicate that a product (eg, any product or a particular product) has been purchased on the website, and that additional profile information corresponding to the route has been provided.

  The interface 600 may include a connection tool 640 for directionally connecting a plurality of icons. Each connection can indicate that the communication decision tree was configured such that the path could extend over the connected nodes in the indicated direction. However, each connection can be conditioned. Thus, the path extends over the connection if the conditions are met. For example, if the condition is met, the decision is made at the branch node (connected to one end of the connection) and the action represented by the communication icon (connected to the other end of the connection) is performed, Connections can be formed. As another example, the condition may be configured such that upon detecting a particular type of interaction associated with the route-related user device, the condition is satisfied.

  Each action definition icon can be associated with one or more action parameters to further define the action to be performed when the route reaches the icon. For example, the parameter definition interface may be displayed as part of interface 600 when the icon is clicked. Also, the parameter definition interface may be configured to open in a pop-up window when the icon is right-clicked and / or double-clicked.

  In some cases, each action definition icon and / or event detection icon corresponds to a widget or code that can be executed independently. The canvas 605 can operate as a communication fabric. Therefore, the result generated by one widget (for example, an indication from the machine learning model corresponding to the switching icon indicating that communication is to be transmitted according to a specific communication specification) is output to another widget (for example, a specific communication specification). (A widget corresponding to a communication icon corresponding to). Therefore, the canvas 605 can extend the route according to the result of the widget, and can adjust the mode of communication exchange.

  Although not shown in FIG. 6, a plurality of switching icons 620 can be arranged on the canvas 605 in some cases. Each switch icon 620 may correspond to an individual instance of a machine learning model that can be individually configured and operated.

  FIG. 7 shows an exemplary parameter definition interface 700 for the switch icon. Parameter definition interface 700 includes a stage label field to accept text input. The text entry is displayed next to the relevant icon in the interface and describes the communication decision tree. A descriptive sentence can also be input via text input. The descriptive sentence is displayed on the interface in response to, for example, detection of a single click or a double click related to the icon, and describes the communication decision tree.

  The parameter definition interface 700 includes fields for defining a target result for a switch icon configured to specify a selection or operating specification and / or a switch icon configured to execute a machine learning model. be able to. For example, a pull-down menu can identify a plurality of events that are being tracked and can be designated as target results. The goal results include actions initiated on the user device, notifications issued by the system, and the like. For example, the target result may be that a link in the communication provided to the user device was clicked, a communication provided to the user device was opened, a purchase (exchange) was made in connection with the user device by communication. And detecting that a chat session has started and that a form has been completed.

  The parameter definition interface 700 may indicate one of the possible results identified for a switch icon configured to specify a selection or operating specification and / or a switch icon configured to execute a machine learning model. These fields can be further included. For example, interface 700 includes fields corresponding to three paths or branches extending from the icon. In this case, a stage label name of another operation definition icon is designated for each route. In some cases, the path information is automatically updated in the parameter definition interface 700 upon detecting that the switch is linked to one or more other icons in the interface for defining a communication decision tree. It should be noted that the parameter definition interface 700 can also include options for adding or deleting additional routes.

  In some cases, one path can be designated as the default path. For example, an additional having a threshold unless the machine learning model predicts that another path has at least a high probability of achieving the desired result or if it passes through another path (eg, indicating improved confidence in subsequent predictions) The path is usually routed to the default path, unless it is expected to generate the same data. In some cases, the selection of a default path depends on the confidence associated with the probability of occurrence of the target result (e.g., another path has at least a 60% probability of producing the target result, and the confidence of that probability is at least 50%). % Will not be selected as the default path unless predicted).

  In some cases, the switch icon may be configured to select a path and / or next action (or lack of action) and a time to extend the path to the next icon (eg, perform the next action). . The time can be selected from multiple times and / or open continuous or closed continuous. In the illustrated example, the parameter definition interface 700 includes a maximum time to extend the path to the next action definition icon. Therefore, unless the decision logic executed in connection with the switch icon indicates that another time period is more advantageous, within one day of the route reaching the switch icon (the probability of obtaining the target result) And / or training data increases), the path needs to be extended.

  Machine learning techniques and / or other selection techniques can be configured to identify a path from a plurality of possible paths that is most likely to produce a target result. In some cases, this technique further introduces some noise and / or variability. Thus, if necessary, a sub-optimal path can be selected to continuously train the underlying model.

  In some cases, the client needs to introduce a bias for the selection of a particular path. For example, a particular path may be more expensive (eg, computationally and / or financially) than another path. As another example, a particular path may be more available than another path. As yet another example, a client may want to quickly obtain information about the validity of a given path to signal a subsequent resource allocation decision.

  Accordingly, the parameter definition interface 700 can include one or more options for biasing individual paths. In the example shown, a slider is provided for each path. Placing the slider on the “increase” side on the right allows the path selection to be adjusted to increase the bias applied to the corresponding path. Placing the slider on the left “constrained” side allows the path selection to be adjusted to reduce the bias on the corresponding path. Restrictions can be imposed on such augmentation and / or restriction options. Therefore, for example, even if the slider is moved to the leftmost constraint position, selection of the corresponding path is not prevented. Such restrictions allow the machine learning model to continue collecting data related to various options and to change one or more parameters through learning. If there are only two options, a single interface component can be provided to specify the relative bias on one option relative to the other. On the other hand, if there are more than two options, the increase / constrained options can provide more intuitive controls depending on the options.

  FIG. 8 illustrates another parameter definition interface 800 that includes options for biasing the display of various content in the communication. The illustrated example shows nine content items (each representing a corresponding product). A slider associated with the visual representation of each content item is provided. Placing the slider on the “increase” side on the right side increases the bias on the corresponding content item (eg, corresponding to selecting multiple paths representing different content or selecting a content after identifying a communication channel). Can be adjusted. Placing the slider on the left “constrained” side allows the content selection to be adjusted to reduce the bias on the corresponding item.

  In the illustrated example, if the slider is located at the leftmost position, a “do not provide” option is displayed. In some cases, if the Do Not Provide option is not selected, at least the first content item is selected.

  Based on the relative bias indicated by the slider and past communication counts, the system can estimate the number of times to display an individual content item on a given day. Thus, as the client moves one or more sliders, the interface 800 can automatically update the estimated count for the number of times an individual content item is displayed (eg, daily) according to the slider position.

  It should be understood that different types of bias can be further specified and provided. For example, one or more sliders can be provided to indicate a bias in relation to the time of transmitting a communication. For example, the slider may indicate the degree of bias that will affect the decision for immediate transmission (and / or transmission at another time, eg, completion time).

  Providing a bias (e.g., for the type of communication channel, display of a particular type of content, or transmission of a communication at a particular time) may include, for example, weighting one or more machine learning models and / or one or more thresholds. It can include making changes. In some cases, biasing performs post-processing (eg, redistributing portions of the results to improve the degree to which the distribution of communication attributes matches the indicated target distribution based on the bias). Including.

  FIG. 9 illustrates yet another parameter definition interface 900 that includes options for biasing the use of various communication channels for transmitting communications. In the example shown, three communication channels are shown: email, app-based notification, and SMS message. Sliders are provided that are visually related to the representation of each channel. Placing the slider on the “increase” side on the right can adjust the content transmission to increase the bias on the use of the corresponding type of channel. Placing the slider on the left “constrained” side allows the path selection to be adjusted to reduce the bias on the corresponding channel.

  Interface 900 further illustrates a time-series representation of the number of communications transmitted using each channel during a recent period and the number of communications scheduled to be transmitted over each channel during a future period. The current time is represented by a vertical line. Communication may be scheduled according to a selection from a number of possible transmission times (eg, which may be included in the same or different machine learning models associated with the selection of the communication channel). Thus, the client can see the expected load across the various channels and can decide whether to adjust the bias applied to the channels.

  FIG. 10 is a flowchart illustrating a process 1000 for specifying the direction of a path through a communication decision tree using machine learning, according to some embodiments of the present invention. The process 1000 begins at block 1005. At block 1005, a data structure representing a communication decision tree is invoked. The communication decision tree can be configured to dynamically define individual paths through the communication decision tree using a machine learning technique to indicate a series of communication specifications. More specifically, the communication decision tree can include a plurality of nodes. A particular path can be extended between nodes in response to detecting an event indicating that a particular path needs to be extended. An event may include, for example, detecting a particular type of action or communication from a user event, or identifying a particular decision (corresponding to identifying a node) in a path management system or machine learning data platform. it can. The plurality of nodes can include a plurality of branch nodes. Each branch node of the plurality of branch nodes is configured to identify an operation point configured to determine a direction of a specific path or a specific operation initiated by a path management system or a machine learning data platform. May correspond to the operating point. The branch node can be configured to specify a direction or an action using the configured machine learning model.

  At block 1010, it is detected that a path (associated with a particular user and / or a particular user device) has been extended to reach a branch node of the communication decision tree. The particular user may be any of a communication recipient (e.g., having one or more predetermined attributes identified by the client) or a plurality of users included in a target group of target customers. The target groups of the communication recipients or target customers can be statically defined, but need not be statically defined. For example, the target group may be a dynamic set corresponding to profiles representing each of one or more predetermined attributes at various times. The path may be a result of detecting a particular type of event initiated at the user device (eg, a communication indicating that the user device is participating in a client-related website session, a communication indicating that the user has submitted a profile form). And / or may be extended to a branch node according to the result of completing a particular system startup operation.

  At block 1015, learning data generated by processing other user data is obtained. Other user data may correspond to data associated with at least a portion of a target group of communication recipients and / or target customers. The learning data may include data generated when training a machine learning technique. The training may be performed at a separate time from directing one or more paths using machine learning techniques, or the training and utilization of machine learning techniques may be performed simultaneously. Other user data may include path data associated with one or more paths through the same or different communication decision trees. For example, other user data may include, for at least a portion of a target group of communication recipients, a target of a communication decision tree designated as a path target with a corresponding path predefined (eg, indicating a successful workflow). It can indicate that the node has been reached. The target node may represent, for example, a response to an interaction, exchange, or communication with the content. As another alternative or additional example, other user data may indicate that for at least a portion of the target group, the corresponding route is an undesirable result (not responding to communications, not exchanging, or interacting with content). May have been reached. Other user data may indicate profile data and / or attributes corresponding to one or more users, and may indicate various events detected and / or initiated in connection with individual paths. Thus, if various situations exist, other user data may indicate the probability of detecting a particular type of event (e.g., specified as a target result by the client).

  At block 1020, one or more user attributes associated with the user corresponding to the route (detected as extending to a branch node) are obtained. User attributes include, for example, the type of user device, the geographical location of the user device, the type of browser used for the user device, the operating system used for the user device, one of the interactions between the user device and a particular website. Part or all history, interaction of the user device with one or more other websites, cookie data associated with the user device, notification type opened on the user device or the type of notification that resulted in the execution of the link (email, text message And / or application messages). One or more specific user attributes may be collected and / or obtained locally and / or requested and received from a remote source.

  At block 1025, one or more communication specifications are identified based on the learning data and one or more user attributes. For example, the learning data can include one or more parameters of a machine learning model (eg, a regression model). Also, a machine learning model can be further defined based on one or more hyperparameters. Thereafter, the machine learning model can be configured to process the user attributes using the parameters, hyperparameters and / or infrastructure. The results of running the model may identify the most possible option to achieve the target result from the plurality of available options. The plurality of available options may correspond to, for example, different types of communication channels used, different types of content transmitted, and / or different transmission times. In some cases, multiple available options share one or more other communication specifications.

  At block 1030, the content is transmitted to a user device associated with the route. The transmission of the content is performed according to one or more communication specifications.

  At block 1035, it is determined whether the path has been extended to reach another branch node in the communication decision tree. The determination is, for example, determining whether or not a threshold time has elapsed since the last communication was transmitted to the specific user (or the corresponding device). And / or determining whether the target content was the result of the last communication sent to a particular user (or corresponding device). , Determining whether a particular user has interacted with the target content. In some cases, each of these two or more decisions is associated with different branch node criteria. Block 1035 may include identifying other branch nodes that the route has reached.

  If it is determined that the path has been extended to reach another branch node, the process 1000 returns to block 1015 and repeats blocks 1015-1035. However, the iterative execution of block 1015 includes obtaining different training data generated by processing other user data (eg, but not necessarily, in combination with at least some user data). The different learning data may be generated using the same or different configurations of machine learning techniques (eg, having the same or different values and / or the same or different types of parameters and / or hyperparameters). The iterative execution of block 1020 may include obtaining at least one other user attribute. The iterative execution of block 1025 may include identifying at least one other communication specification (from a different set of possible communication specifications) based on different learning data and at least one other user attribute. At least one other communication specification may be specified using the same or a different type of base model. Repeated execution of block 1030 may include triggering transmission of other content according to at least one other communication specification.

  If it is determined that the path has not been extended to reach another branch node, the process 1000 proceeds to block 1040 and determines whether the path has been completed. This determination can be made by determining whether the current end of the route is connected to a route that does not have an extended connecting end. If it is determined that the route has been completed, the route processing ends. If it is determined that the path is not complete, the process 1000 returns to block 1035 and determines whether the path has reached another branch node (eg, as a result of a user-initiated action or an external event). can do.

  Accordingly, the process 1000 facilitates using differently configured machine learning models for iteration to identify specifications corresponding to different stages of the communication exchange. At various stages, the model may be based on various profile data (eg, different field values or values that change over time) and / or (eg, various inputs and / or outputs associated with the model and / or Various model parameters (learned based on temporary changes) can be used. Iterative execution of a machine learning model can dynamically indicate the communication exchange of individual users.

  FIG. 11 is a flowchart illustrating a process 1100 for defining a machine learning-based communication decision tree using an interface that supports configurable visual elements. Process 1100 begins at block 1105. At block 1105, an interface that includes a plurality of visual elements and a canvas for arranging the elements is utilized. Each of the plurality of visual elements can be arranged on a canvas. For example, configuring the interface so that the user can place the visual element in another location by clicking on the representation of the visual element and dragging the cursor to another location on the canvas while holding the click Can be. As another example, by selecting an expression (e.g., by clicking or double-clicking) and making another input (e.g., another click or double-click) when the cursor moves to another position, the visual element It can be located at other locations.

  The plurality of visual elements can include a plurality of action defining visual elements. Each of the plurality of action-defined visual elements may be a specific action to be performed when a specific path extends to the action-defined visual element. The plurality of action definition visual elements may include a switching visual element representing a decision action (for example, performed using a machine learning model) for specifying a communication specification using a machine learning technique. The plurality of operation definition visual elements may further include a plurality of communication visual elements. Each of the plurality of communication visual elements may represent a particular communication specification (eg, type of communication channel, particular content and transmission time). In addition, the plurality of visual elements can include a connected visual element configured to connect the plurality of arranged visual elements in a directional manner. Each of the arranged plurality of visual elements may correspond to each of the plurality of operation-defined visual elements. The directional connection may indicate the order in which a particular action represented by the arranged visual elements occurs.

  At block 1110, an update of the canvas is detected. The updated canvas includes a switching visual element located at a first location on the canvas, a first communication visual element of the plurality of communication visual elements located at a second location on the canvas, and a third communication visual element located on the canvas. A second communication visual element of the plurality of communication visual elements arranged at the location may be included. The first communication visual element may represent a specific first communication specification, and the second communication visual element may represent a specific second communication specification.

  The updated canvas may further include a plurality of connected visual elements. Each of the plurality of connected visual elements can include an instance of the connected visual element. A first connection of the plurality of connected visual elements may be arranged to connect the switching visual element to the first communicating visual element. A second connection of the plurality of connected visual elements may be arranged to connect the switching visual element to the second communicating visual element. The plurality of connected visual elements are such that a possible result obtained by performing the machine learning technique on the switching visual element is that a first result that triggers a communication transmission having a specific first communication specification, And a second result that triggers a communication transmission having a communication specification of 2.

  At block 1115, a particular communication decision tree is defined based on the updated canvas. At block 1120, it is detected that the predetermined path associated with the particular profile data has been extended to a particular decision action represented by the switching visual element. In response to this detection, at block 1125, a machine learning result is generated by processing the particular profile data using machine learning techniques (constructed using the learning parameter data and / or the static data). . The learning parameter data includes data learned during individual or continuous training of the machine learning model based on multiple paths associated with other users and / or multiple paths associated with the same or different communication decision trees. be able to. Processing the specific profile data using the machine learning technique can indicate a communication specification applied to content transmission among the first and second specific communication specifications.

  Accordingly, at block 1130, the content is transmitted to the user device associated with the route. The transmission is performed according to one of the specific first and second communication specifications indicated in the machine learning result. For example, the first and second communication visual elements may correspond to different types of communication channels. Thus, block 1125 may include identifying one of two types of communication channels, and the content may be transmitted over the identified channel.

  Thus, the canvas facilitates defining the configuration of the communication decision tree. However, the client need not define a communication exchange that applies to all users and / or a communication exchange that includes only one or more deterministic rules. Rather, the interface generally supports selection of various communication specifications, specification of the order and / or constraints of communication events. Specific communication specifications can be automatically and dynamically generated using machine learning techniques. In this way, the communication system can be easily configured to follow client priorities, and the communication system can be dynamically adapted to the characteristics of a particular user, resource load, recent interaction patterns, and the like.

  It should be understood that variations of the disclosed technology are also contemplated. For example, the branch node can select a communication specification to be used for communication using another type of artificial intelligence model that is not a machine learning model. As another example, the interface can be configured to accept a selection of a particular type or a normal type of artificial intelligence model to be used for the path stage corresponding to the switching element. As yet another example, the interface may include one or more placed on the canvas (e.g., at points corresponding to one or more communication decision trees, one or more time periods, and / or one or more user group segments). It can be configured to show data used for training a machine learning model corresponding to a plurality or all of the switching elements.

  It should be understood that various types of decision trees can be supported using the techniques disclosed herein. For example, in some cases, nodes in the decision tree and / or visual elements on the canvas may correspond to elements related to logic. These logics satisfy certain conditions (e.g., detecting a particular type of device-to-device communication, indicating that an application not associated with the client has performed an operation, or that a certain amount of time has elapsed). If it is detected whether or not the condition is satisfied, a specific operation is executed. For a subset of nodes and / or visual elements, a specific conditional operation is to execute a machine learning model based on profile data to connect nodes (or visual elements) from nodes (or visual elements). Visual element). This allows other specific actions related to the selected node (or visual element) to be performed. For example, machine learning-based routing can be integrated into an "If This Then That" environment. For example, a branch may specify various applications used to store data instead of coupling the branch node to a node that specifies a communication specification. Thus, a decision framework can be established and artificial intelligence applets and / or plug-ins can communicate with one or more other applets and can communicate back through the canvas.

  In some cases, techniques may relate to identifying a worker sending a request to perform a particular job. This technique may additionally or alternatively include, for example, a worker sending a request to participate in a query process and / or one or more requests (eg, performing a particular job and / or participating in a query process). Identifying the configuration can include.

  In some cases, the identification can be made in response to iterations proceeding through the decision tree. The decision tree does not need to include finite decisions and / or consistent possible decisions at each decision node. For example, the decision tree can include a progressively sorted decision tree to iteratively reduce multiple workers at each of two or more decision nodes. The number of possible decisions may correspond to the number of all possible workers reduced in response to a particular decision. Although not required, a particular decision node may constrain or define the number of workers included in the reduced plurality of workers (eg, reducing the plurality of workers to a subset that includes five workers). , Reducing the number of workers to 80%, or reducing the number of workers to a subset that includes 2 to 9 workers). The final output of the decision tree includes, for example, the identification of a single worker, a fixed number of workers, and / or workers that are not numerically constrained.

  A decision tree can include, for example, one or more chance nodes that can reflect responses received from one or more workers. For example, sending and responding (or not responding) to a worker according to the execution of the decision node may indicate whether the worker has accepted the request. In some cases, if a worker rejects and / or rejects the request (eg, within a predetermined time period), the worker is removed from the pool of workers currently under consideration.

  In some examples, the decision tree represents a decision corresponding to a job offer (e.g., a notice to accept employment corresponding to a particular position and / or occupation and / or a particular employer). A job offer is associated with one or more documents, files or datasets that describe a job, ideal candidates, a timeline of decisions, salaries, locations, and the like. Each worker may include one or more particular jobs, one or more particular types of jobs and / or resumes for applying for all jobs and / or identifiers corresponding to the person who submitted the application. . The one or more first decision nodes correspond to decisions regarding workers that employers, recruiters, etc. invite to a query process (eg, a face-to-face interview, a telephone interview, or a web interview). One or more second decision nodes correspond to decisions regarding a worker providing a job offer (eg, a request for a job). One or more third decision nodes correspond to identifying a request configuration (eg, salary, vacation, location, and / or other details of a job offer) for performing the job. As a specific example, a first decision node may be configured to output a ranking of all workers corresponding to a probability of providing an interview for a job, and a second decision node may be configured to output a probability of providing a job. May be configured to output the ranking of all (or a part of) the workers corresponding to. As another specific example, the first decision node may be configured to generate an output corresponding to a likelihood that a worker will accept a job offer notification (and not accept a job offer despite a possible offer configuration). Yes, and the second decision node can be configured to generate the necessary content (details to define salaries, bonuses, and / or rewards) to attract worker interest. The outputs of the decision nodes can be presented or transmitted at the same time or at different times.

  It should be understood that, although not necessary, the decision can be made automatically at each decision node as the predetermined path progresses. For example, rather than selecting all (or a portion thereof) workers to a particular number, an automated process (eg, depending on the execution of a machine learning model) may rank all workers and / or score each worker. Can be output. By presenting and / or transmitting the output (e.g., to a user device), the user can facilitate a decision that indicates the actual screening and / or selection. By communicating the selection and / or selection (eg, in response to user input provided to the portal interface), the path of the decision tree can be further adjusted.

  Sorting, ranking and / or scoring may be based on one or more optimal variables. The one or more optimal variables may be used consistently for multiple or all decisions in the decision tree and may be specifically defined for each decision node (eg, some or all May be different for decision nodes). The optimal variable may correspond to a decision made by the worker (eg, the probability that the worker will accept a request to participate in a query process and / or the probability that a worker will accept a request to perform a job) and determine for the next request. (Probability of sending a request to participate in a query process and / or probability of sending a request to perform a job) and / or capability (eg, a worker keeps executing a job at predetermined time intervals) Probabilities, probabilities of achieving a high reputation of worker performance by third parties, and / or probabilities of securing high rewards or bonuses based on job performance).

  In some examples, ranking of workers (e.g., a subset filtered from an initial set of workers) is performed based on performance data. For example, a machine learning model can be trained based on training data that includes worker attributes and performance data. Thus, the attributes and job data of each of a plurality of workers (eg, in a subset filtered from the initial set of workers) can be input to the model. The output may indicate a ranking of the plurality of workers based on an estimated ability rating of each of the plurality of workers (e.g., an estimated ability corresponding to an estimated ability evaluation when assigning workers to jobs). Outputs include, for example, numerical ranking (eg, a number assigned to each worker), order (eg, a particular order of workers), and numerical or categorical scores for each worker (eg, ordering workers based on these scores). Can be attached). By training the model, for example, one or more weights that associate various worker attributes with estimation capabilities can be learned. In some cases, the model can be trained to adjust the capacity estimates based on job specifications in addition to worker attributes.

  The performance data used to train the model can include, for example, numerical data, categorical data, and / or textual data. In some cases, performance data may include metrics along the scale and / or processing the performance data may generate metrics along the scale. Performance data is received from one or more job owners (eg, an employer or interviewer) at one or more times (eg, at regular times, such as annually, in response to pull requests or push reports). be able to. The training data may include attributes of the worker at the time (or near that time) of assigning the job whose performance is being evaluated to the worker and / or attributes of the current time of the worker (eg, the current time with respect to the time during which the performance is being evaluated). Can be included. The attribute of a specific time can be confirmed based on a specification data set that can be used by a worker at a specific time. In some cases, the saved specification data set can be updated based on the passage of time to derive attributes used for training. For example, if the specification data set includes an age-experience attribute and that attribute was first saved or last updated at time tx (e.g., if a worker is executing a job associated with x hours) The attribute can be updated by adding x to the attribute (assuming). In some cases, performance data is included in the training dataset only if one or more relevant worker attributes have been updated within a predetermined time period.

  FIG. 12 is a diagram illustrating an exemplary progressive screening decision tree 1200, according to some embodiments of the present invention. A decision tree can evaluate multiple workers associated with a particular job. In some cases, the reputation includes a reputation for a plurality of worker files (each representing a particular worker and / or corresponding to a particular worker) and / or a job file (eg, representing a job and / or a job offer). The worker file may represent or include a worker specification data structure (eg, resume data). The job file may represent or include a job specification data structure (eg, job posting data).

  In the decision node 1205, the worker included in the worker set is determined. This determination can be made based on an evaluation of a plurality of files or data representations corresponding to the plurality of workers and a file or data representation corresponding to the job. Specifically, each file or data representation (corresponding to a worker or job) can be processed and converted into a feature data set. For example, the feature data set can include an array, matrix, or vector that includes a value at each element. The value may indicate the degree to which the corresponding file or data representation matches the particular feature, identifies the particular feature, or corresponds to the particular feature.

  In some cases, each feature is defined to be a particular space within a word space. As a simple example, a feature may represent a cluster of words (which may include, for example, synonyms and / or other corresponding words), and assign a weight to each word (although not necessarily). it can. As another example, a feature can represent a distribution. Words may be identified based on, for example, semantic similarity to other words and / or probabilities of occurring in a document, file, paragraph, field value, and / or sentence similar to one or more other words. Can be placed throughout the dimensional word space. The distribution can be defined, for example, according to the center or centroid and weight (and / or distortion) of each of one dimension, multiple dimensions or all dimensions.

  The feature data set can be generated using data from a specification data structure that can include unstructured data (eg, words having a variable length in terms of number of characters or words). By processing this data, individual words to be further processed can be identified. For example, further processing may include converting each word to a stem (eg, by removing the endings such as "ing", "ed", etc.). Further, various words (eg, stopwords, prepositions, connectives, etc.) can be deleted from the set of words.

  By inputting the (eg, processed) word set into a machine learning model or other artificial intelligence model, a feature data set can be identified. As an example, a feature data set can be calculated using a machine learning model including a neural network language model. In some cases, the relative order of words in the corresponding worker specification data structure affects feature assignment. An exemplary type of machine learning model is the report by Le, Quoc, published in JMLR: W & CP Vol. 32, in the bulletin of the 31st International Conference on Machine Learning, Beijing, China, 2014. Distributed Representation ". The article is available from https://cs.stanford.edu/~quocle/paragraph_vector.pdf and is incorporated herein by reference in its entirety for all purposes. For example, doc2vec can be used to convert a text input containing a set of words into a feature data set.

  As another example, the value of a feature may be calculated, for example, by generating a weight for each word in the processed word set based on the particular space of the feature and averaging the weights. As yet another example, the value of a feature is determined by counting the number of occurrences of a group of words associated with the feature in the data from the specification data structure (and counting this count based on other counts or the total number of words in the data). By normalizing).

  Individual feature data sets (eg, for individual workers or individual jobs) can be stored. Each feature data set corresponds to a worker or job identifier and / or other information (eg, one or more key-value pairs from a corresponding specification data structure, other information from a corresponding specification data structure, a corresponding Raw data from the specification data structure). In some cases, the feature dataset and related information can be stored in a Solr database.

  Determining the workers included in the worker set includes comparing a feature data set of each worker (e.g., included in the plurality of workers) with a feature data set of the job, and determining a portion of the plurality of workers based on the comparison. This may include selecting a set. The plurality of workers may include, for example, workers identified as having potential jobs (eg, in response to submission of a specification data structure). The plurality of workers may be dynamic. That is, a worker can be dynamically deleted from multiple workers (eg, in response to receiving a job assignment or deletion request for a worker) and / or (eg, upon receipt of a new worker specification data structure). T) Workers can be added dynamically from multiple workers.

  Performing a comparison between the worker feature data set and the job feature data set may include generating a comparison metric based on the feature data set. Metrics can include, for example, angles, dot products, correlation coefficients, and the like. A subset of workers may be defined as including workers associated with dot products or correlation coefficients (or other comparison metrics) that exceed an absolute or relative threshold. The relative threshold can be configured to identify a subset that includes a certain number or a certain percentage of workers.

  At decision node 1210, a plurality of workers are ranked by repetition of the machine learning model, feature dataset, and additional information. In other words, the machine learning model can be executed to process inputs corresponding to the worker feature dataset, the job feature dataset, and the additional information. The additional information may include one or more key-value pairs from a worker specification dataset (eg, representing or indicating years of experience, education level, etc.). In some cases, the additional information may additionally or alternatively include one or more key-value pairs from the job specification data set.

  Thus, in some cases, the machine learning model used in the early stages does not receive and / or process some of the worker data received and / or processed in the later stages by the machine learning model. For example, a late-stage machine learning model may receive and / or process one or more key-value pairs for various workers, while an early-stage machine learning model may include one or more keys for various workers. Do not receive and / or process value pairs. The same applies to the opposite case. For example, early-stage machines receive and / or process raw text, while late-stage machine learning models receive and / or process raw text, rather than raw text.

  Using different types of machine learning models, different types of inputs and / or different optimal variables at different stages of the decision tree iteration has technical advantages. In some cases, the use of different types of inputs can facilitate efficient processing. For example, by using processed worker data in place of the original data at a later stage, past data that can be performed to highlight or select useful (or potentially useful) information without repeating the processing Processing and learning are available. Also, the early-stage processing can learn from a larger and / or more diverse data set (although not required), so if it relies on early-stage learning over late-stage learning, The dimension reduction process performed is more robust.

  In some cases, by selectively using some data at a later stage, a rich worker dataset can be utilized while promoting efficient processing and resource consumption. For example, a large worker population may be associated with tens of thousands or millions of key-value pairs. When considering each additional category of key-value pairs, the amount of space for possible worker attributes increases exponentially. On the other hand, the size of a possible worker pool may decrease gradually, so the absolute impact of including one or more key-value pairs in the later stages is less than in the earlier stages. In other words, in some cases, the efficiency and practicability of performing these steps on a computer can be improved by designing the early-stage calculations relatively simpler than the late-stage calculations.

  The machine learning model can include a model configured to output a ranking of various inputs (given a reference job specification data structure for a job feature data set). For example, a machine learning model may use a ranking learning algorithm.

  Machine learning models can be implemented to use supervised learning, semi-supervised learning, or reinforcement learning. As an example, a machine learning model can use LambdaRank (by using an xgboost implementation). The machine learning model can be trained using communications indicating the degree to which different worker data (eg, worker specification data structures) correspond to different job data (eg, job specification data structures). Note that other types of machine learning models that do not output the ranking of each worker may be used. For example, a machine learning model may include, for some or all of a plurality of workers, worker data (eg, a worker feature data set and / or additional information associated with a worker) and job data (eg, a job feature data set and / or Alternatively, an output indicating a matching score indicating the degree of matching with the additional information related to the job may be generated.

  In some cases, the execution output of the machine learning model can be further processed. For example, yet another process includes identifying a subset of workers associated with a ranking below a predetermined ranking threshold and / or identifying a subset of workers associated with a matching score above a predetermined score threshold. Can be included.

  At decision node 1215, a subset of the plurality of workers is determined. The determination may include identifying a worker sending a request to participate in the query process by filtering the workers based on a ranking (or other result) from the decision node 1210. In some cases, the decision is made automatically. For example, a rule may indicate that the subset includes a predetermined number of workers with the lowest (highest) ranking or that each worker associated with a score that exceeds a predetermined threshold is included in the subset.

  In some cases, a decision is made based on receiving a communication corresponding to a decision of a worker included in the subset. The communication may be generated based on and in response to input received at a user device (eg, a recruiter's device). For example, determining a ranking at node 1210 may generate a communication indicating a ranking associated with a corresponding worker. The communication may be configured to cause the interface to be updated and / or created and to be displayed on the user device. The interface includes some or all of the ranking. The interface may further include and / or utilize information regarding the ranked workers. For example, the interface may include links for browsing or displaying on-screen the worker's specification dataset, spatially associated with each ranking. The interface may include one or more input options (eg, checkboxes, pull-down menus, links, drag and drop interfaces or radio buttons). Interactions with options can be automatically detected and translated into the determination of particular workers included in the subset.

  As an example, the interface can include a table where each row represents a different worker. One column indicates the ranking of the worker and another column may include other information about the worker (eg, field and value pairs, worker ID). A split element (eg, a movable thick line) can be used to indicate a worker (eg, a worker above the split element) included in the subset. The interface can include drag and drop elements for moving and reordering rows. As another example, the interface can include a visual object (eg, a small box) that identifies worker-specific information and worker rankings. Each visual object can include a checkbox and can be spatially related to the checkbox. Checking the box indicates that the worker is included in the subset.

  In some cases, the request to participate in the query process is automatically sent to each worker in the subset (eg, mailed or displayed to a worker's portal in the job assignment system). In some cases, participates in the query process by displaying or sending identification information of the workers in the subset (along with key-value information for identifying the contacts of the workers in the subset, e.g., email addresses or telephone numbers). To facilitate external communication of requests. The request may include, for example, job information (eg, job title and employer), a location (eg, address) associated with the query process, and a time (or time option) for the query process.

  In some cases, the response to the request can be tracked. For example, if a request is sent via email or a worker portal, provide one or more response options to indicate whether the worker accepts the request and / or whether the worker rejects the request. be able to. As another example, a communication may be received from a user indicating a response associated with each worker in the subset. In some cases, a determination that a response has not been received from a particular worker within a predetermined period of time (eg, from the time the request was sent) may generate a virtual response that the request was rejected. By further filtering the subset, certain workers that have rejected the request can be removed.

  In some cases, the decision node 1210 may not be included in the decision tree 1200. Identifying the workers included in the subset can be performed using machine learning model iterations. The machine learning model includes a worker specification data structure (e.g., corresponding to a plurality of workers), a job specification data structure, and additional information (e.g., key-value pairs associated with the plurality of workers and / or keys associated with the job). And value pairs). The machine learning model can be configured to output, for example, rankings, scores, or binary identifiers (corresponding to being included in a subset).

  A decision node 1220 determines one or more job configurations for the job request, and a decision node 1225 determines a worker that sends a request to execute the job. In various examples, the determination of node 1220 and the determination of 1225 are made simultaneously. Alternatively, the node 1220 is determined before the node 1225 is determined. Alternatively, the node 1220 is determined after the node 1225 is determined. In some cases, one or both of the decisions made in blocks 1220 and 1225 are made using a machine learning model.

  For block 1220, the machine learning model is sufficient (and / or minimal) for each worker, partial worker, or all workers of the worker's subset to cause the worker to accept the (configured) job. Is configured to output one or more job configurations estimated to be The model can be further configured, if desired, to output a result indicating that the worker is assumed not to accept the job regardless of the configuration.

  For block 1225, the machine learning model (same or different) is configured to generate output related to a selection of one or more workers to send a request to perform the job. The output may include, for example, as one or more output values, rankings, scores and / or estimates, estimated job capability values, and / or estimated maintenance probabilities or time periods associated with the likelihood of accepting the offer. Can be. The decision of the worker sending the request to perform the job may be made automatically and / or based on communications reflecting the decision made by the remote system and / or the user. Good. For example, various data, such as ranking, score or minimum specification data, may be displayed on an interface that accepts a selection of one or more workers in a subset of the workers sending the request. Each request may, but need not, be configured with the corresponding configuration information identified in block 1220. In some cases, the request is configured with a default (or minimum or maximum, as consistent) configuration.

  Thus, decision tree 1200 illustrates a method of iteratively refining a worker pool using a plurality of decision nodes to assign jobs. In some cases, responses to job requests are tracked (eg, as described above with respect to possible approaches to tracking responses to requests that participate in a job query process). If one or more or all of the job requests are rejected (eg, and / or not accepted within the default period), delete the previously selected worker under consideration and associate with blocks 1215, 1220 and 1225 One or two of the decisions to be made can be repeated.

  FIG. 13 is a diagram illustrating an exemplary representation of data processing that occurs during the implementation of a progressive screening decision tree, according to some embodiments of the present invention.

  In the illustrated example, at a first decision node, a transform element 1305 (eg, a code block configured to transform one or more input data objects into processed data objects) includes a plurality of worker files 1310 and 110. Receive one or more jobs. Each worker file 1310 can include a worker specification data structure that includes a plurality of worker specifications. Each job file 1315 can include a job specification data structure that includes a plurality of job specifications. In some cases, conversion element 1305 may not need to receive a worker filer and / or job file, but may instead identify worker specification data and / or job specification data to be converted. For example, specification data is received via one or more communications generated and transmitted from individual worker devices and / or other devices in response to detecting input (eg, corresponding to an online form entry). You may.

  The conversion element 1305 can be configured to convert each worker file 1310 into a worker feature data set 1320 and each job file 1315 into a job feature data set 1325. Each worker feature data set 1320 and each job feature data set 1325 may include a value indicating the degree to which each feature of the feature set is indicated in a corresponding (job or worker) file. Each feature may correspond to a portion of semantic space 1330. In some cases, the machine learning model may define each feature using a training data set (eg, including unstructured data). Machine learning models can be trained via unsupervised learning and can include, for example, doc2vec. Features may be defined based on co-occurrence of words and / or phrases, and may be defined based on structural analysis (eg, word grouping, sentence and / or paragraph detection). In some cases, the optimal variables used during training (to identify the feature set) may include orthogonality metrics between features. That is, the feature set may be defined to maximize the degree to which each individual feature is orthogonal and / or independent of other features.

  The comparison element 1335 can generate a comparison output of each combination pair by comparing each worker feature data set 1320 and each job feature data set 1325. In some cases, the comparison output includes a cosine angle between the feature data sets. In some cases, the comparison generates a correlation coefficient between the feature data sets, and generates a difference metric (eg, mean, median, maximum, or minimum) based on feature differences between the feature data sets. Including.

  The first filter 1335 can determine a subset 1340 of the worker feature data set 1320 for each job feature data set 1325 by processing the comparison output. This processing includes, for example, determining a predetermined number of workers from the plurality of workers based on the comparison output generated for the plurality of workers. For example, the culling may include a comparison output having the 100 highest values (a higher value indicates strong similarity. Alternatively, if the comparison output is set such that a lower value indicates strong similarity, (A comparison output having the lowest value may be determined). In some cases, filtering includes selecting each worker feature dataset associated with a comparison output that exceeds a predetermined threshold (or, in an alternative output configuration, below a predetermined threshold).

  At a second decision node, the machine learning model 1345 may further process the subset 1340 of the worker feature data set 1320 and each job feature data set 1325. The machine learning model 1345 can be trained via supervised learning. Machine learning model 1345 can be configured to process structured data (eg, having a predetermined format and / or size). The machine learning model 1345 is configured to output, for each of the subsets 1340 of the worker feature data set, a value (for example, ranking or score) indicating the degree of correspondence with the job related to the job feature data set 1325. be able to. The output may include a ranking 1350 of each worker of one, some or all of the subsets 1340. The machine learning model may include a learning-to-rank model. In some cases, machine learning model 1345 is trained based on optimal variables corresponding to input received from a user. The input may identify past decisions (eg, whether one or more particular workers have been offered an interview and / or whether one or more particular workers have accepted the interview, To determine whether one or more particular workers have accepted the job, and / or to determine an evaluation of job performance).

  In addition to the feature data set, the machine learning model 1345 may further process data from the worker file 1310 corresponding to the subset 1340 and / or the job file 1315 corresponding to the job feature data set 1325. Other data may include, for example, key and value data and / or individual form entries and may be identified by a second filter 1355 implemented. In some cases, other data for a particular worker (or job) can be aggregated (eg, concatenated) into a corresponding worker feature data set 1340 (or job feature dataset 1325), and the concatenated data can be Can be entered.

  Ranking 1350 may be processed, for example, by a third decision node 1355, which may further filter and reduce (e.g., to a predetermined size) a plurality of worker subsets 1340. In some cases, further screening involves identifying a predetermined number of workers with the lowest ranking values. In some cases, other information (eg, feature data sets, key-value pairs) may be further used. In some cases, the ranking is presented to the user and further sorting corresponds to a selection of one or more workers indicated by one or more communications from the user device. In some cases, further screening includes (e.g., past workers who have requested to participate in the query process, past workers who have accepted participation in the query process, past workers selected based on job performance, accepted jobs, etc.). (Using past workers and / or past job performance as optimal variables). Further, the filtered subset of workers includes, for example, workers sending requests to participate in a query process. If the request is rejected or not answered, the third decision node 1355 may be configured to identify another worker in the further filtered subset.

  At a fourth decision node 1360, the worker subset is further evaluated by evaluating the results from the query process (e.g., to identify a predetermined number of workers to perform the job, e.g., a single worker). You can also sort out. This determination is made by a machine learning model (using past workers selected based on job performance, past workers who accepted jobs and / or past job performance as optimal variables). Further, the selected subset of workers includes, for example, workers sending requests to execute jobs. If the request is rejected or not answered, fourth decision node 1355 may be configured to identify another worker that is included in the further filtered subset.

  At a fifth decision node 1365, job specification information for each worker can be decided. For example, one or more job specifications corresponding to a lower threshold that is estimated to be sufficient for each of the one or more workers to accept a job can be identified. In some cases, the decision nodes 1360 and 1365 may be merged. For example, threshold identification data for each worker may be identified and transmitted to a user device, which may further respond to selection of a worker included in the filtered subset.

  Each decision made at the third decision node 1355, the fourth decision node 1360, and the fifth decision node 1365 can be fed back to the machine learning model 1345. Additional feedback 1370 may include, for example, whether an individual worker has accepted a request to participate in a query process and / or has received a request to perform a job and / or execute a job associated with the worker. Quality can be shown.

  FIG. 14 is a flowchart illustrating a process 1400 for advancing a path through a decision tree using a machine learning model iteratively according to some embodiments of the present invention.

  Process 1400 begins at block 1405. At block 1405, a data structure (e.g., a data file or dataset) is invoked that represents a decision tree for assigning jobs. One or more first decision nodes of the decision tree may correspond to a decision on how to process and convert unstructured data of worker information or job information into meaningful structured data. Transformations include, for example, unsupervised machine learning trained to detect various relationships and structures between words, and then define a feature set used to characterize unstructured data (eg, text data) It can be performed using a model.

  At block 1410, a job feature data set is generated using a machine learning model. The job feature data set can include a series of numerical values that indicate the characteristics of the job (via a display indicating the degree to which the various characteristics displayed in the job information are specified). The job feature data set can be generated, for example, by processing job information (eg, job offer data) received via an online portal.

  At block 1415, a set of worker specification data for each of the plurality of workers is generated using the machine learning model. The worker feature data set can include a series of numerical values that indicate the characteristics of the worker. The worker feature data set can be generated, for example, by processing worker information (eg, resume data) received via an online portal. The job feature data set can be generated by performing a semantic analysis of the worker information.

  The specification data set (for individual workers and / or jobs) may be generated by a single decision node, or the worker specification data set and the job specification data set may be separated (eg, based on different learning parameters). May be generated by the decision node.

  At block 1420, it is detected that the path corresponding to the iteration of the decision tree has reached another decision node. Other decision nodes may, for example, correspond to decisions related to how to screen (or screen) a plurality of workers and / or how to compose a request sent to the workers. Other decision nodes may make decisions based on processing the input data using a machine learning model (e.g., a different type of model than the model used to identify features of the feature dataset). , May be configured. The machine learning model can include, for example, a neural network model.

  At block 1425, a learning parameter corresponding to the node is obtained. The learning parameters can include one or more weights of the neural network. The learning parameters may be generated during the training process.

  At block 1430, the machine learning model is iteratively executed. A machine learning model can be configured using the node-specific learning parameters. Execution includes inputting the job-related feature dataset and one or more or all worker-related feature datasets into a machine learning model. In some cases, each worker-related feature dataset is processed with the job-related feature dataset, but not with other worker-related feature datasets (the other worker-related feature datasets are individually processed with the job-related feature dataset). It is processed). In some cases, all worker-related feature data sets and job-related feature data sets are processed by a particular execution of the model. The results of the model include, for example, selecting a plurality of workers, ranking a plurality of workers, and assigning a score to each of the plurality of workers.

  If the decision tree includes additional decision nodes, the process 1400 may return to block 1420 and use the machine learning model to make or facilitate the decision. This machine learning model may be the same type as the model used for the decision node, or may be a different type.

  At a block 1435, one or more communication operations are performed that represent one or more results generated using the machine learning results generated at the one or more decision nodes. For example, the communication may include a request sent to a worker participating in a query process or accepting a job. As another example, the communication may include identifying one or more workers to be included in the subset of workers generated at the decision node. As yet another example, the communication may include, for each of the one or more workers, e.g., the degree to which the worker specification corresponds to the job specification, the degree to which the worker will accept requests to participate in the query process, It may include a ranking or score generated using a machine learning model that represents the estimated degree of accepting a request to perform a job, or the estimated degree of a worker successfully executing a job.

  The specific details described above are provided to provide a thorough understanding of the embodiments. It is to be understood that embodiments may be practiced without these specific details. For example, circuits may be disclosed in block diagrams in order not to obscure the embodiments in unnecessary detail. As another example, well-known circuits, processes, algorithms, structures, and techniques have not been disclosed in detail to avoid obscuring the embodiments.

  The techniques, blocks, steps, and means described above can be implemented in various ways. For example, these techniques, blocks, steps, and means may be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units may include one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (PLDs). (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to perform the functions described above, and / or combinations thereof.

  Also, it should be noted that each embodiment has been described as a process shown as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart describes the operations as a sequential process, many operations can be performed in parallel or concurrently. Further, the order of the operations may be rearranged. The process ends when the operation is completed, but may include additional steps not shown in the figure. Processing can correspond to methods, functions, procedures, subroutines, subprograms, and the like. If the operation corresponds to a function, its termination may correspond to a return of the calling function or the main function.

  Further, embodiments may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium, such as a storage medium. A code segment or machine-executable instruction can represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transferring and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be communicated, transferred, or transmitted via any suitable means, such as memory sharing, message transfer, token transfer, network transmission, and the like.

  For a firmware and / or software implementation, the methods may be implemented with modules (eg, procedures, functions, and so on) for performing the functions described herein. Any of the machine-readable media tangibly embodied in the instructions may be used to implement the methods described herein. For example, software code can be stored in memory. The memory may be implemented inside the processor or external to the processor. The term “memory,” as used herein, refers to any type of long-term, short-term, volatile, non-volatile, or other storage media, and refers to a particular type of memory. The invention is not limited to a specific number of memories or types of media storing the memories.

  Further, as disclosed herein, the term “storage media” refers to read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash It may include one or more memories for storing data, including memory devices and / or other machine-readable media for storing information. The term “machine-readable medium” includes removable and fixed storage, optical storage, wireless channels, and / or various other storage media that can contain or carry instructions and / or data, It is not limited to these.

  While the principles of the present disclosure have been described above with reference to particular apparatus and methods, it should be understood that this description is illustrative only and is not limiting upon the scope of the present disclosure.

Claims (20)

A computer implemented method,
Invoking a data structure representing a decision tree for assigning a job, wherein the decision tree includes a plurality of consecutive decision nodes and is configured to perform iterative decisions corresponding to progressive selection of workers. Extending each of the paths to each of the plurality of consecutive decision nodes triggers an intermediate or final decision on the configuration of a worker request to execute the job,
Initiating a path of the decision tree, wherein the path is associated with a plurality of workers that may be assigned to the job;
Detecting that the path has reached a first decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a first set of node-specific learning parameters generated by training a first machine learning model using the first set of training data, wherein the first set of node-specific learning parameters comprises: Define how to convert unstructured data into structured feature data,
Invoking a plurality of sets of worker-related unstructured data, wherein each of the plurality of sets of worker-related unstructured data corresponds to each worker of the plurality of workers and includes unstructured data;
Performing one or more first iterations of the first machine learning model configured using the first set of node-specific learning parameters using the plurality of sets of worker-related unstructured data. Performing the one or more first iterations of the first machine learning model generates a first result including a plurality of sets of worker feature data, wherein the first result corresponds to each worker of the plurality of workers. And each of the plurality of sets of worker feature data, in which the worker-related unstructured data set indicates a degree of correspondence corresponding to the worker, represents each of a set of features, and each of the plurality of sets of worker feature data is: Structured data, wherein the plurality of features are defined based on the first set of node-specific learning parameters;
Identifying a subset of the plurality of workers for continuous evaluation for the job based on the plurality of sets of worker feature data;
Detecting that the path has reached a second decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a second set of node-specific learning parameters generated by training a second machine learning model using the second set of training data, wherein the second set of node-specific learning parameters comprises: Define how to convert structured data,
Performing one or more second iterations of a second machine learning model configured using the second set of node-specific learning parameters using the subset of the plurality of sets of worker feature data. Executing each of the one or more second iterations of the second machine learning model, wherein each of the subsets of the plurality of sets of worker feature data corresponds to each worker of the subset of the plurality of workers. Generating a second result indicative of an estimated absolute or relative correspondence between the job and each worker of the at least one subset of the plurality of workers;
Triggering a communication operation representative of said second result. Further comprising identifying a plurality of key-value pairs,
Each of the plurality of key-value pairs corresponds to a respective worker of the subset of the plurality of workers;
Performing the one or more second iterations of the second machine learning model further uses the plurality of key-value pairs;
The method of claim 1, wherein performing one or more first iterations of the first machine learning model does not use the multiple key-value pairs. Learning the first set of node-specific learning parameters using an unsupervised learning process;
2. The method of claim 1, further comprising learning the second set of node-specific learning parameters using a supervised learning process. Each worker feature data set of the plurality of sets of worker feature data includes a vector, a matrix, or an array;
The method of claim 1, wherein the vector, matrix or array has a predetermined number of dimensions. Calling an unstructured job dataset corresponding to the job;
Performing another iteration of the first machine learning model configured with the first set of node-specific learning parameters using the unstructured job data set. Performing the further iteration of the machine learning model generates a job feature data set indicating a degree to which the unstructured job data set represents each of the set of features;
Generating a job comparison metric for each worker of the plurality of workers based on the worker feature data set and the job feature data set corresponding to the worker, wherein the subset of the plurality of workers includes: The method of claim 1, wherein the method is identified based on the job comparison metric generated for the plurality of workers. Further comprising: training the first machine learning model to learn the first set of node-specific learning parameters using the first set of training data;
The first set of training data includes other unstructured data;
The method of claim 1, wherein the first set of node-specific learning parameters associates certain semantic characteristics with experience levels, skills, or job characteristics. Detecting that the path has reached a third decision node of the plurality of consecutive decision nodes of the decision tree;
Using at least one of the plurality of sets of worker feature data to perform a third iteration of a third machine learning model configured using a third set of node-specific learning parameters; A third result of the execution of the third iteration of the third machine learning model is that one or more request configurations estimated to result in acceptance of a request by a particular worker of the subset to execute a job. The method of claim 1, wherein the request is represented and configured using the one or more request configurations.   The method of claim 1, wherein triggering the communication operation comprises transmitting a communication including the second result to a user equipment. One or more workers requesting to participate in a query process are identified from the plurality of workers based on the second result,
The method of claim 1, wherein the method further comprises sending a request to participate in the query process to a device associated with each of the one or more workers.   The method of claim 1, wherein the second result comprises a ranking or score of each worker of the subset of the plurality of workers. Detecting that the path has reached a second decision node of the plurality of consecutive decision nodes of the decision tree;
Identifying a particular worker giving the request to perform the job;
2. The method of claim 1, further comprising: identifying the particular worker or triggering another communication sent to the particular worker. A computer program product tangibly embodied on a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform the following operations, said operations comprising:
Invoking a data structure representing a decision tree for assigning a job, the decision tree including a plurality of consecutive decision nodes, and configured to execute an iterative decision corresponding to a progressive selection of workers. Extending each of the paths to each of the plurality of consecutive decision nodes triggers an intermediate or final decision on the configuration of a worker request to execute the job,
Initiating a path of the decision tree, wherein the path is associated with a plurality of workers that may be assigned to the job;
Detecting that the path has reached a first decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a first set of node-specific learning parameters generated by training a first machine learning model using the first set of training data, wherein the first set of node-specific learning parameters comprises: Define how to convert unstructured data into structured feature data,
Invoking a plurality of sets of worker-related unstructured data, wherein each of the plurality of sets of worker-related unstructured data corresponds to each worker of the plurality of workers and includes unstructured data;
Performing one or more first iterations of the first machine learning model configured using the first set of node-specific learning parameters using the plurality of sets of worker-related unstructured data. Performing the one or more first iterations of the first machine learning model generates a first result including a plurality of sets of worker feature data, wherein the first result corresponds to each worker of the plurality of workers. And each of the plurality of sets of worker feature data, in which the worker-related unstructured data set indicates a degree of correspondence corresponding to the worker, represents each of a set of features, and each of the plurality of sets of worker feature data is: Structured data, wherein the plurality of features are defined based on the first set of node-specific learning parameters;
Identifying a subset of the plurality of workers for continuous evaluation for the job based on the plurality of sets of worker feature data;
Detecting that the path has reached a second decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a second set of node-specific learning parameters generated by training a second machine learning model using the second set of training data, wherein the second set of node-specific learning parameters comprises: Define how to convert structured data,
Performing one or more second iterations of a second machine learning model configured using the second set of node-specific learning parameters using the subset of the plurality of sets of worker feature data. Executing each of the one or more second iterations of the second machine learning model, wherein each of the subsets of the plurality of sets of worker feature data corresponds to each worker of the subset of the plurality of workers. Generating a second result indicative of an estimated absolute or relative correspondence between the job and each worker of the at least one subset of the plurality of workers;
A computer program product, comprising triggering a communication operation representative of the second result. The operation further includes identifying a plurality of key-value pairs,
Each of the plurality of key-value pairs corresponds to a respective worker of the subset of the plurality of workers;
Performing the one or more second iterations of the second machine learning model further uses the plurality of key-value pairs;
13. The computer program product of claim 12, wherein performing one or more first iterations of the first machine learning model does not use the multiple key-value pairs. The operation is
Learning the first set of node-specific learning parameters using an unsupervised learning process;
13. The computer program product of claim 12, further comprising: learning the second set of node-specific learning parameters using a supervised learning process. Each worker feature data set of the plurality of sets of worker feature data includes a vector, a matrix, or an array;
13. The computer program product according to claim 12, wherein the vector, matrix or array has a predetermined number of dimensions. The operation is
Calling an unstructured job dataset corresponding to the job;
Performing another iteration of the first machine learning model configured with the first set of node-specific learning parameters using the unstructured job data set. Performing the further iteration of the machine learning model generates a job feature data set indicating a degree to which the unstructured job data set represents each of the set of features;
Generating a job comparison metric for each worker of the plurality of workers based on the worker feature data set and the job feature data set corresponding to the worker, wherein the subset of the plurality of workers includes generating The computer program product of claim 12, wherein the computer program product is identified based on the job comparison metrics of the plurality of workers. The operation further comprises training the first machine learning model to learn the first set of node-specific learning parameters using the first set of training data;
The first set of training data includes other unstructured data;
13. The computer program product of claim 12, wherein the first set of node-specific learning parameters associates particular semantic characteristics with experience levels, skills, or job characteristics. The operation is
Detecting that the path has reached a third decision node of the plurality of consecutive decision nodes of the decision tree;
Using at least one of the plurality of sets of worker feature data to perform a third iteration of a third machine learning model configured using a third set of node-specific learning parameters; A third result of the execution of the third iteration of the third machine learning model is that one or more request configurations estimated to result in acceptance of a request by a particular worker of the subset to execute a job. 13. The computer program product of claim 12, wherein the request is represented and configured using the one or more request configurations.   13. The computer program product of claim 12, wherein triggering the communication operation comprises sending a communication including the second result to a user device. The system
One or more data processors;
A non-transitory computer readable storage medium containing instructions, wherein the instructions, when executed on the one or more data processors, cause the one or more data processors to perform the following operations: Is
Invoking a data structure representing a decision tree for assigning a job, the decision tree including a plurality of consecutive decision nodes, and configured to execute an iterative decision corresponding to a progressive selection of workers. Extending each of the paths to each of the plurality of consecutive decision nodes triggers an intermediate or final decision on the configuration of a worker request to execute the job,
Initiating a path of the decision tree, wherein the path is associated with a plurality of workers that may be assigned to the job;
Detecting that the path has reached a first decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a first set of node-specific learning parameters generated by training a first machine learning model using the first set of training data, wherein the first set of node-specific learning parameters comprises: Define how to convert unstructured data into structured feature data,
Invoking a plurality of sets of worker-related unstructured data, wherein each of the plurality of sets of worker-related unstructured data corresponds to each worker of the plurality of workers and includes unstructured data;
Performing one or more first iterations of the first machine learning model configured using the first set of node-specific learning parameters using the plurality of sets of worker-related unstructured data. Performing the one or more first iterations of the first machine learning model generates a first result including a plurality of sets of worker feature data, wherein the first result corresponds to each worker of the plurality of workers. And each of the plurality of sets of worker feature data, in which the worker-related unstructured data set indicates a degree of correspondence corresponding to the worker, represents each of a set of features, and each of the plurality of sets of worker feature data is: Structured data, wherein the plurality of features are defined based on the first set of node-specific learning parameters;
Identifying a subset of the plurality of workers for continuous evaluation for the job based on the plurality of sets of worker feature data;
Detecting that the path has reached a second decision node of the plurality of consecutive decision nodes of the decision tree;
Obtaining a second set of node-specific learning parameters generated by training a second machine learning model using the second set of training data, wherein the second set of node-specific learning parameters comprises: Define how to convert structured data,
Performing one or more second iterations of a second machine learning model configured using the second set of node-specific learning parameters using the subset of the plurality of sets of worker feature data. Executing each of the one or more second iterations of the second machine learning model, wherein each of the subsets of the plurality of sets of worker feature data corresponds to each worker of the subset of the plurality of workers. Generating a second result indicative of an estimated absolute or relative correspondence between the job and each worker of the at least one subset of the plurality of workers;
Triggering a communication operation representative of said second result.