WO2022235404A1 - Composing human-readable explanations for user navigational recommendations - Google Patents
Composing human-readable explanations for user navigational recommendations Download PDFInfo
- Publication number
- WO2022235404A1 WO2022235404A1 PCT/US2022/024687 US2022024687W WO2022235404A1 WO 2022235404 A1 WO2022235404 A1 WO 2022235404A1 US 2022024687 W US2022024687 W US 2022024687W WO 2022235404 A1 WO2022235404 A1 WO 2022235404A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- vector
- recommendation
- sequence
- user interface
- human
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/954—Navigation, e.g. using categorised browsing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
- G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/045—Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
Definitions
- the present disclosure relates to composing human-readable explanations for user navigational recommendations.
- the present disclosure relates to using a product of a neural network, that generates a user navigational recommendation, for a composing human- readable explanation for the user navigational recommendations.
- Various user interface applications provide recommendations to users that the users can select to navigate to a service.
- the application may indicate the reason for the navigational recommendations. For example, the application may indicate the user has made a similar selection in the recent past or other similar users have made the recommended selection.
- applications include only a few categories of reasons that can be provided to a user as an explanation for a recommendation. A limited, static set of reasons are typically programmed by a developer.
- Figure 1 illustrates a system in accordance with one or more embodiments
- Figure 2 illustrates an example set of operations for generating a human-readable reason for a user interface navigational recommendation in accordance with one or more embodiments
- Figure 3 illustrates an example set of operations for training a machine learning model to generate a human-readable reason for a user interface navigation in accordance with one or more embodiments
- Figure 4 shows a block diagram that illustrates a computer system in accordance with one or more embodiments.
- a navigation platform provides users access to multiple different applications and services.
- Each application and service may include many different functions.
- Each of these functions is a potential navigational target that a user may navigate to within the navigation platform to perform the function.
- One or more embodiments compose human-readable explanations (also referred to herein as “reasons”) for navigational recommendations.
- Composing a human-readable explanation includes individually selecting words or phrases that are then analyzed, combined, rearranged, modified, or removed to generate the human-readable explanation for a navigational recommendation .
- the system generates a navigational recommendation by applying a trained neural network to a set of conditions.
- the set of conditions may include, for example, prior user activity, a current position in the navigation platform, a current day and time.
- the system In applying the trained neural network, the system generates an initial vector that encodes the set of conditions.
- the initial vector also referred to herein as a “layer”
- additional vectors additional “layers”
- a final vector that may be mapped to the navigational recommendation
- one of the vectors generated by the application of the neural network, for generating the navigational recommendation is extracted for use in composing the human-readable explanation for the navigational recommendation.
- the vector, generated by the neural network and extracted for use in composing the human-readable explanation may be referred to herein as a “recommendation vector”.
- a decoder trains a machine learning model to generate the human-readable reasons for the navigational recommendations based on (1) historical recommendation vectors, and (2) historical human-readable reasons associated with the recommendation vectors.
- the system generates a dictionary of human-readable reasons for recommendations, with each entry of the dictionary including: (1) a recommendation identifier (ID) associated with a recommended navigational target, (2) a reason identifier (ID) associated with a particular reason for the recommendation, and (3) a human-readable reason associated with the reason ID.
- the machine learning model that generates the human-readable reasons for the navigational recommendations is a sequence-to-sequence (seq2seq) machine learning model.
- the machine learning model generates the human- readable reasons for navigational recommendations by receiving the recommendation vector as an input and generating an initial word as an output.
- the recommendation vector and the initial word are provided to the machine learning model as inputs to generate a next word in the human- readable reasons for the navigational recommendation.
- the process is repeated until each word in the human-readable reasons for the navigational recommendation is generated.
- the words generated by the machine learning model may be represented as embedded vectors that are associated with words or phrases.
- the decoder identifies the words and phrases associated with the embedded vectors to generate the human-readable reasons for the navigational recommendations .
- the machine learning model may generate any number of human-readable reasons for user navigational recommendations that are customized to a particular user. For example, two users may be provided with different reasons for a same user interface navigational recommendation. In addition, the same user may be provided with different reasons for a recommendation at different times. Since the human-readable reasons are generated by the machine learning model, the number of human-readable reasons is not limited to reasons entered by operators into the system. In addition, operators are not required to enter particular rules to display particular human-readable reasons for particular recommendations. Instead, the machine learning model generates a customized human-readable reason for a customized recommendation based on sets of user attributes and sets of navigational target attributes.
- Figure 1 illustrates a system 100 in accordance with one or more embodiments.
- system 100 includes a navigation platform 110, an encoder 120, a decoder 130, and a data repository 110.
- the navigation platform 110 may be a program or set of programs providing access to a suite of applications and services.
- the navigation platform 110 includes a user interface 111 and navigational targets 112.
- user interface 111 refers to hardware and/or software configured to facilitate communications between a user and the navigation platform 110.
- Interface 111 renders user interface elements and receives input via user interface elements.
- interfaces include a graphical user interface (GUI), a command line interface (CLI), a haptic interface, and a voice command interface.
- GUI graphical user interface
- CLI command line interface
- haptic interface a haptic interface
- voice command interface a voice command interface
- user interface elements include checkboxes, radio buttons, dropdown lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.
- interface 111 different components of interface 111 are specified in different languages.
- the behavior of user interface elements is specified in a dynamic programming language, such as JavaScript.
- the content of user interface elements is specified in a markup language, such as hypertext markup language (HTML) or XML User Interface Language (XUL).
- the layout of user interface elements is specified in a style sheet language, such as Cascading Style Sheets (CSS).
- interface 111 is specified in one or more other languages, such as Java, C, or C++.
- Navigational targets 112 include applications, documents, and executable files. Examples of navigational targets include: a program to review an invoice in an accounting application; a program to store, edit, and display an expense report in an accounting application; a program to store, edit, and display customer and lead contact information in a customer acquisition application; a program to retrieve sales data, and to generate and display a sales forecast in a business operations application; a program to store, edit, and display workflow information of projects in a project management application; a program to store, edit, and display employee information in a human resources application; and a program to retrieve and display compliance data for compliance with government regulations. While the navigational targets, above, are provided by way of example, these are not an exhaustive list of possible navigational targets.
- Embodiments encompass any program and any application to which a user may navigate to retrieve information and/or perform a function in a navigation platform.
- the navigation platform 110 generates a user interface element in a GUI of the user interface 111 to allow a user to select the user interface element to access a navigational target 112.
- Each navigational target 112 is associated with an application 113. Many different navigational targets may belong to the same application.
- an accounting application includes navigational targets corresponding to different functions in the accounting application, including: generating expense reports; generating financial forecasts; generating payroll information; etc.
- a human resources application includes navigational targets that include: view employee list; view and edit employee information; request leave; generate employee performance reports; view organization; etc.
- Each navigational target 112 includes navigational target attributes.
- the navigation platform 110 displays in the GUI of the user interface 111 a set of user interface elements to allow a user to navigate to a corresponding set of navigational targets 112.
- the displayed user interface elements may belong to navigational targets 112 associated with different applications 113.
- the displayed user interface elements may belong to different documents or executable files within the same application.
- the encoder 120 obtains user attribute data 141 and navigational target attribute data 142 from the data repository 140.
- the user attribute data 141 and navigational target attribute data 142 may be obtained from the navigation platform 110.
- a user may log in to the navigation platform 110 and generate a user profile including user attribute data.
- User attribute data 141 include data about particular users. Examples of user attribute data 141 include: job name, role, assignment, location, and business unit. While the user attributes, above, are provided by way of example, these are not an exhaustive list of possible user attributes. Embodiments encompass any information that may be associated with a user to identify attributes of the user.
- Navigational target attribute data include data about the navigational target.
- navigational target attributes include: application stripe, product family, view ID, menu ID, taskflow ID, WebAppName, contextualAreaWidth, RecentltemsID, ObjectCode, CreationDate, LastUpdateLogin, LastUpdateDate, EnterpriselD, ParentID, SourceFile. While the navigational target attributes, above, are provided by way of example, these are not an exhaustive list of possible navigational target attributes. Embodiments encompass any information that may be associated with a navigational target to identify attributes of the navigational target.
- the data repository 140 is any type of storage unit and/or device (e.g., a file system, database, collection of tables, or any other storage mechanism) for storing data.
- a data repository 140 may include multiple different storage units and/or devices. The multiple different storage units and/or devices may or may not be of the same type or located at the same physical site.
- a data repository 140 may be implemented or may execute on the same computing system as the navigation platform 110, the encoder 120, or the decoder 130. Alternatively, or additionally, a data repository 140 may be implemented or executed on a computing system separate from the navigation platform 110, the encoder 120, or the decoder 130.
- a data repository 140 may be communicatively coupled to the navigation platform 110, the encoder 120, or the decoder 130 via a direct connection or via a network.
- Information describing user attribute data 141, navigational target attribute data 142, historical training data 114, and sorted lists 117 may be implemented across any of components within the system 100. However, this information is illustrated within the data repository 140 for purposes of clarity and explanation.
- the encoder 120 generates an input vector based on user attribute data 141 and navigational target attribute data 142.
- the encoder may convert the user attribute data 141 and navigational target attribute data 142 to numerical values and vector values.
- the encoder may concatenate the converted user attribute data 141 and navigational target attribute data 142 to generate the input vector.
- the encoder 120 provides the input vector to the machine learning engine 121 to generate a recommendation vector 122.
- the machine learning engine 121 may include a trained machine learning model that has been trained to receive the input vector and generate one or more recommended navigational targets based on the input vector.
- the machine learning model is a contextual multi-armed bandit-type machine learning model.
- the encoder 120 provides the recommendation vector 122 to a recommendation engine 123 to generate the one or more recommended navigational targets based on the input vector.
- the encoder 120 may convert the recommendation vector 122 to a probability distribution using a SoftMax layer prior to providing the recommendation vector 122 to the recommendation engine 123.
- the recommendation engine 123 identifies, based on the information in the recommendation vector 122, one or more recommended navigational targets for the user. Each navigational target is represented by a unique recommendation ID 124.
- the recommendation engine 123 may generate an ordered list of recommendation IDs 124 for a corresponding list of recommendation vectors 122.
- the encoder 120 also provides the recommendation vector 122 to the decoder 130 to train another machine learning model to generate human-readable reasons for user interface navigational recommendations.
- the recommendation vector 122 is a multi-dimensional vector having vector values, generated by the machine learning engine 121, based on relationships between user attribute data, navigational target attribute data, a recommended user interface navigation, and a reason for the recommendation.
- the recommendation vector 122 includes contextual information of a particular recommendation associated with a particular reason for the recommendation. Therefore, the recommendation vector 122 includes contextual information of a sequence of words associated with the particular navigational recommendation.
- the recommendation vector 122 including the contextual information of the sequence of words, is provided to a machine learning engine 132 that trains a machine learning model 132a to learn from the contextual information of the sequence of words included in the recommendation vector to sequentially generate a sequence of output vectors 134 corresponding to a human-readable reason for the navigational recommendation.
- the machine learning model 132a is a bi-directional sequence-to- sequence (seq2seq) machine learning model 132a, where an input sequence is encoded within the recommendation vector 122.
- the machine learning model 132a includes a long short-term memory (LTSM) type model, a gate recurrent unit (GRU) type model, or an extended recurrent neural network (RNN) type model.
- LTSM long short-term memory
- GRU gate recurrent unit
- RNN extended recurrent neural network
- the machine learning engine 132 trains the machine learning model 132a to generate a human-readable reason for a recommendation based on (1) the recommendation vector 122, and (2) embedded vectors 131 of words for a reason ID associated with the recommendation vector 122.
- the decoder 130 includes reason ID logic 138 that identifies a reason for the recommendation included in the contextual information of the recommendation vector 122.
- the reason ID logic associates the reason with a reason ID, which is a unique identifier associated with the human-readable reason for the recommendation.
- the recommendation vector 122 is provided as an input vector to the machine learning model 132a to generate a first output vector Yi of a sequence of output vectors 134.
- the first output vector Yi may be generated from among the set of embedded vectors 131 that represent the words for a particular reason ID.
- the first output vector Y i and the recommendation vector 122 are provided as an input vector to the machine learning model 132a to generate the next output vector Y2 of the sequence of output vectors 134.
- Each of the output vectors 134 represents an embedded vector of a word in the sequence of words 134 that make up the human-readable reason for a recommendation.
- the machine learning model 132a repeats the process of providing a previously-generated output vector and the recommendation vector 122 as an input vector to generate a next output vector in the sequence 134 until each output vector Yi- Y n has been generated.
- the decoder 130 then generates the human-readable reason for the recommendation using the sequence 134 of output vectors Yi-Y n .
- the machine learning engine 132 trains the machine learning model 132a based on historical training data 144 stored in the data repository 140.
- the historical training data 144 includes: (1) historical recommendation vectors 145, (2) recommendation IDs 146 associated with the historical recommendation vectors 145, (3) reason IDs associated with the historical recommendation vectors 145, (4) reason attribute data 148, and (5) a text corpus 149.
- the text corpus 116 may include words that have previously been used to generate human-readable reasons for navigational recommendations, as well as words from any other sources, such as a dictionary.
- the historical training data 144 may include one thousand (1,000) reason IDs 147 and an associated one thousand (1,000) sequences of words corresponding to the respective reason IDs 147.
- the decoder 130 converts the sequence of words associated with a particular reason ID 147 for a reason included in a corresponding historical recommendation vector 145 into an embedded vector 131 of words.
- the machine learning model 132a For a predetermined subset of the historical training data 144, the machine learning model 132a generates a sequence of output vectors 134 corresponding to a human-readable sequence of words for a historical reason ID 147 associated with a historical recommendation vector 145.
- the machine learning engine 132 may compare the generated human-readable sequences of words with the sequences of words associated with the historical reason IDs 147 to train the machine learning model 132a.
- the machine learning engine 132 includes an attention engine 135. For each input vector, other than the recommendation vector 122, in the sequence of input vectors 133, the attention engine 135 applies weights to vector values of the recommendation vector 122 to focus the machine learning model 132a on words in the reason for the recommendation.
- the attention engine performs a dot-product operation on (1) the set of vectors Yi-Y n representing the words of a particular reason ID, and (2) the recommendation vector 122, to generate a scalar value.
- the attention engine 135 generates an attention distribution based on the scalar values for the vectors Yi-Y n .
- the attention engine 135 converts the attention distribution to a vector and applies the vector to the recommendation vector 122. Consequently, the machine learning model 132a applies a greater weight to a particular word, represented by a particular embedded vector Yi-Y n among the embedded vectors 131 representing the words associated with the reason ID.
- the decoder 130 includes a beam search engine 136 to improve the efficiency of the generation of output vectors Y2-Y 11 by the machine learning model 132a.
- the beam search engine 136 For each input vector, other than the recommendation vector 122, in the sequence 133 of input vectors Yi-Y n , the beam search engine 136 generates k sets of words identified as having the highest probability of being the next words in the reason for the recommendation.
- the beam search engine 136 may identify four sets of words, each set made up of a sequence of three words, having the highest probability of being the next three words in the sequence of words corresponding to the sequence 134 of output vectors Y2-Y 11 .
- the beam search engine 136 provides the k sets of words to the machine learning model 132a to influence the machine learning model 132a towards generating output vectors Y2-Y 11 based on the sets of words having the highest probability of being the next words in the sequence.
- the machine learning engine 132 may coefficient values for neurons within a neural network to influence the neural network to generate an output vector corresponding to a particular word or set of words.
- the machine learning model 132a generates, for a given input recommendation vector 122, a set 137 of (1) a recommendation ID, corresponding to a recommendation for a navigation target for a user, (2) a reason ID, corresponding to a particular reason for providing the navigational recommendation, and (3) a human-readable reason for the recommendation, including a sequence of words describing the reason for providing the recommendation.
- the words of the human-readable reason may be stored as vector values Yi-Y n that are converted into words by the navigation platform 110.
- the system 100 may generate a sorted list 143 of recommendation IDs, reason IDs, and human-readable reasons for the navigational recommendations.
- the encoder may generate a recommendation vector 122 based on the user attribute data 141 of the user.
- the recommendation engine 123 may generate one or more recommendation IDs 124 based on the recommendation vector 122.
- the navigation platform 110 may obtain human-readable reasons for the navigational recommendations from the sorted list 117, if a corresponding reason ID exists in the sorted list.
- the decoder 130 may generate a new reason ID and a new human-readable reason for a recommendation based on the new reason ID.
- the navigation platform 110 displays, in the user interface 111, user interface elements corresponding to the recommended navigational targets and human-readable reasons for the navigational recommendations.
- encoder 120, decoder 130, and navigation platform 110 refers to hardware and/or software configured to perform operations described herein for composing human-readable reasons for user interface navigational recommendations. Examples of operations for composing human-readable reasons for user interface navigational recommendations are described below with reference to Figure 2.
- one or more of the encoder 120, the decoder 130, and the navigation platform 110 are implemented on one or more digital devices.
- digital device generally refers to any hardware device that includes a processor.
- a digital device may refer to a physical device executing an application or a virtual machine.
- Examples of digital devices include a computer, a tablet, a laptop, a desktop, a netbook, a server, a web server, a network policy server, a proxy server, a generic machine, a function- specific hardware device, a hardware router, a hardware switch, a hardware firewall, a hardware firewall, a hardware network address translator (NAT), a hardware load balancer, a mainframe, a television, a content receiver, a set-top box, a printer, a mobile handset, a smartphone, a personal digital assistant (“PDA”), a wireless receiver and/or transmitter, a base station, a communication management device, a router, a switch, a controller, an access point, and/or a client device.
- PDA personal digital assistant
- the system 100 may include more or fewer components than the components illustrated in Figure 1.
- the components illustrated in Figure 1 may be local to or remote from each other.
- the components illustrated in Figure 1 may be implemented in software and/or hardware. Each component may be distributed over multiple applications and/or machines. Multiple components may be combined into one application and/or machine. Operations described with respect to one component may instead be performed by another component.
- Figure 2 illustrates an example set of operations for composing human-readable reasons for user interface navigational recommendations in accordance with one or more embodiments.
- One or more operations illustrated in Figure 2 may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in Figure 2 should not be construed as limiting the scope of one or more embodiments.
- an encoder generates a user interface navigational recommendation by providing a vector, including user attribute data and navigation attribute data, to a trained machine learning model (Operation 202).
- the encoder may convert user attribute values and navigation attribute data values to numerical values and vector values.
- the encoder may generate an input vector for the trained model based on the converted user attribute values and navigation attribute data values.
- the trained machine learning model is a contextual multi-armed bandit-type machine learning model.
- the encoder Prior to generating the user interface navigational recommendation, the encoder extracts a recommendation vector from the model (Operation 204).
- the recommendation vector includes contextual information of a recommended user interface navigation and a corresponding reason for the recommendation.
- the encoder extracts the recommendation vector prior to converting the recommendation vector into a probability distribution with a SoftMax layer.
- the encoder may then generate a navigational recommendation, and associate the recommendation with a recommendation ID, using the probability distribution.
- the recommendation vector is a hidden layer within the machine learning model that is generated prior to generating a user interface navigational recommendation and associated recommendation
- the encoder supplies the recommendation vector to a decoder (Operation 206).
- the decoder identifies a reason ID associated with the recommendation vector.
- the reason ID corresponds to a human-readable sequence of words that give a reason for a particular navigational recommendation.
- the decoder generates a set of embedded vectors corresponding to the words associated with the reason ID.
- the decoder generates a human-readable reason for a recommendation in the recommendation vector using machine-learning model (Operation 208).
- the machine learning model is a bi-directional deep-learning sequence-to- sequence model.
- the machine learning model may be a long short-term memory (LTSM) type model, a gate recurrent unit (GRU) type model, or an extended recurrent neural network (RNN) type model.
- the machine learning model may receive as an input sequence the contextual information in the recommendation vector and may generate as an output sequence a series of embedded vectors representing words.
- the sequence of words describes the reason for the user interface navigational recommendations associated with the recommendation vector.
- the decoder generates the human-readable reason by providing, as input vectors to the machine-learning model, the recommendation vector and one or more previously-generated output vectors.
- the machine-learning model generates an initial output vector based on the recommendation vector.
- the machine-learning model generates each subsequent output vector based on (1) the recommendation vector and (2) at least one previously-generated output vector.
- the embedded vectors representing words of a reason for a navigational recommendation define a set of vectors from among which the output vectors are generated.
- the machine-learning engine may identify, from among the embedded vectors, the embedded vector having the highest probability of corresponding to the input vector.
- the decoder stores the generated human-readable reason for a recommendation together with a reason ID and a recommendation ID (Operation 210).
- the decoder may generate an ordered list of reasons for user navigational recommendations in which each entry includes (1) a recommendation ID associated with a recommended navigational target, (2) a reason ID associated with the recommended navigational target, and (3) a human-readable reason associated with the reason ID.
- the decoder Prior to receiving the recommendation vector from the encoder, the decoder trains the machine learning model.
- the model is trained based on (1) historical recommendation vectors, and (2) embedded vectors representing words of a reason associated with a reason ID associated with the historical recommendation vectors.
- the system identifies a user profile (212).
- the user profile may be a stored user profile, a newly-generated user profile, a user profile of a user logging into the system, or a user profile generated by any other means.
- the user profile includes attribute information about the user.
- the system may generate sets of navigational recommendations and human-readable reasons for the navigational recommendations for any users that are registered to use the system to access applications or services, without requiring the users to be logged into the system.
- the system may detect a user login and may generate sets of navigational recommendations and human-readable reasons for the navigational recommendations based on detecting the user login.
- the system may generate sets of navigational recommendations and human-readable reasons for the navigational recommendations based on user navigation to a particular graphical user interface.
- Embodiments encompass the system generating sets of navigational recommendations and human-readable reasons for the navigational recommendations based on any predetermined event or trigger.
- the encoder generates a recommendation vector using the user attribute information in the user profile and any navigation attribute information associated with the user (Operation 214).
- the encoder may generate the recommendation vector using the trained contextual multi armed bandit-type machine learning model, for example.
- the system identifies a recommendation ID and a reason ID associated with the recommendation vector. For example, the system may run the recommendation vector through a SoftMax layer to generate a probability distribution. The system may generate a recommendation ID based on the probability distribution. A decoder may identify a reason ID associated with the recommendation vector. The system determines whether a human-readable reason associated with the recommendation ID and the reason ID already exists (Operation 216). In particular, the system may refer to a dictionary storing sets of: (1) a recommendation ID, (2) a reason ID, and (3) a human-readable reason associated with the recommendation, to determine whether the human-readable reason already exists. If a human-readable reason associated with the recommendation ID and the reason ID was previously generated and stored, the system retrieves the human-readable reason from the dictionary storing the human-readable reasons for the navigational recommendations (Operation 218).
- the encoder determines that a human-readable reason has not yet been generated for the recommendation ID and the reason ID
- the encoder provides the recommendation vector to the trained machine learning model of the decoder (Operation 220).
- the trained model sequentially generates the words for the human-readable reason for the recommendation.
- the system may store the set of recommendation ID, reason ID, and human-readable reason associated with the recommendation in the dictionary of previously-generated human-readable reasons for recommendations.
- the system displays the navigational recommendation associated with the recommendation ID together with the human-readable reason for the recommendation associated with the reason ID (Operation 222). For example, a user may access a graphical user interface to navigate to one or more applications or services. The system may display the navigational recommendations and corresponding human-readable reasons for the navigations on the graphical user interface.
- the reason ID associated with a particular sequence of words may include information regarding a ranking for the particular sequence of words. For example, the system may determine that a particular navigational target is recommended based on three separate reason IDs, each associated with the different human-readable reason for the recommendation. The system may select only the highest-ranked recommendation to display the human-readable information for the recommendation. Alternatively, the system may select only a predetermined number of human-readable reasons for a recommendation, such as only two or fewer reasons for a recommendation.
- the recommendation vector may include information restricting the human-readable reasons that may be displayed for particular users having particular attributes.
- An organization may not want its general membership to know that certain classes of users are navigating to navigational targets associated with information that may be confidential or sensitive. For example, if a company is looking to acquire a competitor, a legal department may need to navigate to targets related to the competitor.
- the system may block users having a user attribute “division” with a value other than “legal” from generating recommendation vectors resulting in recommendation IDs associated with the competitor, or human-readable reasons identifying other employees as navigating to the navigational targets associated with the competitor.
- the system may restrict human-readable reasons identifying searches of other users that reveal sensitive information, such as users accessing navigational targets related to salary, retirement, disability, personal leave, parental leave, or medical benefits.
- the system may prevent the generation of human-readable reasons for particular recommendations.
- the system may allow the generation of the human-readable reasons, but the system may assign the stored human-readable reasons with a ranking or designator preventing the use or display of the human-readable reasons with a recommendation. Accordingly, a system administrator may identify the types of recommendations that generate confidential or sensitive human-readable reasons.
- Figure 3 illustrates an example set of operations for training machine learning model to sequentially generate words for human-readable reasons for user interface navigational recommendations in accordance with one or more embodiments.
- One or more operations illustrated in Figure 3 may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in Figure 3 should not be construed as limiting the scope of one or more embodiments.
- a machine learning engine obtains historical user navigation data (Operation 302).
- the user navigation data includes user interface recommendation vectors associated with users and user navigation, historical reason ID’s associated with the navigational recommendations, and human-readable reasons associated with the reason ID’s.
- a navigation platform may give users access to many different applications and many different processes, documents, and functions within each application. Each process, document, and function that may be accessed by a user may be a separate navigational target.
- the recommendation vectors are generated by a trained machine learning engine and include contextual information of historical user navigation data, including information about how often each user accesses each process, document, and function in each application provided by the navigation platform.
- the machine learning engine generates training sets of recommendation vectors, recommendation ID’s, and corresponding human-readable reasons for recommendations (Operation 304).
- the training sets include particular user navigational recommendations, represented by recommendation IDs and associated human-readable reasons, represented by reason IDs.
- the machine learning engine uses the training sets to train a machine learning model to generate human-readable reasons for user navigational recommendations (Operation 306).
- the machine learning engine selects a particular recommendation vector corresponding to a particular human-readable reason for the recommendation (Operation 308).
- the machine learning engine converts the words of the human-readable reason into embedded vectors (Operation 310).
- the machine learning engine includes a pre-trained word- to-vector (word2vec) model having been trained with all the words in a text corpus including the words of all the human-readable reasons for all the user navigational recommendations in the historical data.
- the machine learning engine converts the words of the particular reason for the recommendation into the vectors generated with the word2vec model.
- the machine learning engine inputs the recommendation vector into another machine learning model to generate an initial output vector (Operation 312).
- the machine learning model may be a bi-directional sequence-to-sequence (seq2seq) machine learning model.
- the model may be a long short-term memory (LTSM) type model, a gate recurrent unit (GRU) type model, or an extended recurrent neural network (RNN) type model.
- LTSM long short-term memory
- GRU gate recurrent unit
- RNN extended recurrent neural network
- the initial output vector is selected from among the embedded vectors representing the words associated with the historical human-readable reason for the recommendation associated with the recommendation vector.
- the initial output vector may be selected from among a combined set of: (1) the words associated with the historical human-readable reason for the recommendation, and (2) a set of words identified as being related, in the word2vec model, to the words in the historical human-readable reason.
- the machine learning engine takes the initial output vector and the recommendation vector and inputs the two vectors into the human-readable reason-generating model to generate the next word in the sequence of output vectors representing human-readable words of a reason for a recommendation (Operation 314).
- the machine learning engine takes the previously- generated output vector and the recommendation vector, and inputs the two vectors into the model to generate the next output vector (Operation 316).
- the machine learning engine includes an attention engine.
- the attention engine applies weights to vector values of the recommendation vector to focus the human-readable reason-generating machine learning model on a particular next word in the reason for the recommendation (Operation 320).
- the attention engine performs a dot-product operation on (1) the embedded vectors representing the words associated with the historical human-readable reason for the recommendation, and (2) the recommendation vector associated with the recommendation, to generate a scalar value.
- the attention engine generates an attention distribution based on the scalar values for the embedded vectors.
- the attention engine converts the attention distribution to a vector and applies the vector to the recommendation vector. Consequently, the machine learning engine applies a greater weight to a particular word, represented by a particular embedded vector among the embedded vectors representing the words associated with the recommendation.
- the machine learning engine also includes a beam search engine.
- the beam search engine At each stage in which an output vector and the recommendation vector are applied as inputs to the human-readable reason-generating machine learning model, the beam search engine generates k sets of output vectors, corresponding to words in a reason for a recommendation, identified as having the highest probability of being the next output vector (Operation 322).
- the beam search engine may identify four sets of output vectors, each set made up of a sequence of three output vectors, having the highest probability of being the next output vectors in the sequence of output vectors generated by the machine learning model.
- the beam search engine provides the k sets of output vectors to the machine learning engine to influence the machine learning model towards generating particular output vectors having the highest probability of being the next output vectors in the sequence.
- the machine learning engine repeats the process of providing a previously-generated output vector together with the recommendation vector as inputs to the machine learning model to generate a next output vector until every word in the human-readable reason for a recommendation has been generated (Operation 318).
- the generated output vectors are converted into their corresponding sequence of words, and the sequence of words is output to a user.
- the user may provide feedback regarding the sequence of words (Operation 324).
- the user may identify a word that does not make sense among the sequence of words, words that are out of order, or words that are either unnecessary in the sequence or missing from the sequence.
- the machine learning engine may provide to the user one or more alternate words based on relationships identified between words by the word2vec model.
- the user may indicate that a particular human-readable reason should not be generated based on a particular user having particular user attributes. For example, a reason may reveal confidential or sensitive information about other users’ navigations.
- the machine learning engine updates the machine learning model based on the user feedback (Operation 326). For example, if a user indicates words as being in an incorrect order, the machine learning engine may adjust the attention engine to increase a probability of a particular word occurring in a particular position. If a user indicates one or more words as being omitted from the sequence, the machine learning engine may add one or more output vectors to the sequence. If the user indicates a word is unnecessary in the sequence, the machine learning engine may omit an output vector from the sequence. If a user indicates that a particular reason should not be generated, the machine learning model may refrain from generating a human- readable reason for a particular recommendation vector.
- a computer network provides connectivity among a set of nodes.
- the nodes may be local to and/or remote from each other.
- the nodes are connected by a set of links. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, an optical fiber, and a virtual link.
- a subset of nodes implements the computer network. Examples of such nodes include a switch, a router, a firewall, and a network address translator (NAT). Another subset of nodes uses the computer network.
- Such nodes also referred to as “hosts” may execute a client process and/or a server process.
- a client process makes a request for a computing service (such as, execution of a particular application, and/or storage of a particular amount of data).
- a server process responds by executing the requested service and/or returning corresponding data.
- a computer network may be a physical network, including physical nodes connected by physical links.
- a physical node is any digital device.
- a physical node may be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a generic machine that is configured to execute various virtual machines and/or applications performing respective functions.
- a physical link is a physical medium connecting two or more physical nodes. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, and an optical fiber.
- a computer network may be an overlay network.
- An overlay network is a logical network implemented on top of another network (such as, a physical network).
- Each node in an overlay network corresponds to a respective node in the underlying network.
- each node in an overlay network is associated with both an overlay address (to address to the overlay node) and an underlay address (to address the underlay node that implements the overlay node).
- An overlay node may be a digital device and/or a software process (such as, a virtual machine, an application instance, or a thread)
- a link that connects overlay nodes is implemented as a tunnel through the underlying network.
- the overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.
- a client may be local to and/or remote from a computer network.
- the client may access the computer network over other computer networks, such as a private network or the Internet.
- the client may communicate requests to the computer network using a communications protocol, such as Hypertext Transfer Protocol (HTTP).
- HTTP Hypertext Transfer Protocol
- the requests are communicated through an interface, such as a client interface (such as a web browser), a program interface, or an application programming interface (API).
- HTTP Hypertext Transfer Protocol
- the requests are communicated through an interface, such as a client interface (such as a web browser), a program interface, or an application programming interface (API).
- HTTP Hypertext Transfer Protocol
- API application programming interface
- a computer network provides connectivity between clients and network resources.
- Network resources include hardware and/or software configured to execute server processes. Examples of network resources include a processor, a data storage, a virtual machine, a container, and/or a software application.
- Network resources are shared amongst multiple clients. Clients request computing services from a computer network independently of each other.
- Network resources are dynamically assigned to the requests and/or clients on an on- demand basis.
- Network resources assigned to each request and/or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and/or (c) the aggregated computing services requested of the computer network.
- Such a computer network may be referred to as a “cloud network.”
- a service provider provides a cloud network to one or more end users.
- Various service models may be implemented by the cloud network, including but not limited to Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a- Service (IaaS).
- SaaS Software-as-a-Service
- PaaS Platform-as-a-Service
- IaaS Infrastructure-as-a- Service
- SaaS a service provider provides end users the capability to use the service provider’s applications, which are executing on the network resources.
- PaaS the service provider provides end users the capability to deploy custom applications onto the network resources.
- the custom applications may be created using programming languages, libraries, services, and tools supported by the service provider.
- IaaS the service provider provides end users the capability to provision processing, storage, networks, and other fundamental computing resources provided by the network resources. Any arbitrary applications, including an operating system, may be deployed on the network resources.
- various deployment models may be implemented by a computer network, including but not limited to a private cloud, a public cloud, and a hybrid cloud.
- a private cloud network resources are provisioned for exclusive use by a particular group of one or more entities (the term “entity” as used herein refers to a corporation, organization, person, or other entity).
- entity refers to a corporation, organization, person, or other entity.
- the network resources may be local to and/or remote from the premises of the particular group of entities.
- cloud resources are provisioned for multiple entities that are independent from each other (also referred to as “tenants” or “customers”).
- the computer network and the network resources thereof are accessed by clients corresponding to different tenants.
- Such a computer network may be referred to as a “multi-tenant computer network.”
- Several tenants may use a same particular network resource at different times and/or at the same time.
- the network resources may be local to and/or remote from the premises of the tenants.
- a computer network comprises a private cloud and a public cloud.
- An interface between the private cloud and the public cloud allows for data and application portability. Data stored at the private cloud and data stored at the public cloud may be exchanged through the interface.
- Applications implemented at the private cloud and applications implemented at the public cloud may have dependencies on each other. A call from an application at the private cloud to an application at the public cloud (and vice versa) may be executed through the interface.
- tenants of a multi-tenant computer network are independent of each other.
- a business or operation of one tenant may be separate from a business or operation of another tenant.
- Different tenants may demand different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, Quality of Service (QoS) requirements, tenant isolation, and/or consistency.
- QoS Quality of Service
- tenant isolation and/or consistency.
- the same computer network may need to implement different network requirements demanded by different tenants.
- tenant isolation is implemented to ensure that the applications and/or data of different tenants are not shared with each other.
- Various tenant isolation approaches may be used.
- each tenant is associated with a tenant ID.
- Each network resource of the multi-tenant computer network is tagged with a tenant ID.
- a tenant is permitted access to a particular network resource only if the tenant and the particular network resources are associated with a same tenant ID.
- each tenant is associated with a tenant ID.
- Each application, implemented by the computer network is tagged with a tenant ID.
- each data structure and/or dataset, stored by the computer network is tagged with a tenant ID.
- a tenant is permitted access to a particular application, data structure, and/or dataset only if the tenant and the particular application, data structure, and/or dataset are associated with a same tenant ID.
- each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular database.
- each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular entry.
- the database may be shared by multiple tenants.
- a subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is permitted access to a particular application only if the tenant ID of the tenant is included in the subscription list corresponding to the particular application.
- network resources such as digital devices, virtual machines, application instances, and threads
- packets from any source device in a tenant overlay network may only be transmitted to other devices within the same tenant overlay network.
- Encapsulation tunnels are used to prohibit any transmissions from a source device on a tenant overlay network to devices in other tenant overlay networks.
- the packets, received from the source device are encapsulated within an outer packet.
- the outer packet is transmitted from a first encapsulation tunnel endpoint (in communication with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (in communication with the destination device in the tenant overlay network).
- the second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device.
- the original packet is transmitted from the second encapsulation tunnel endpoint to the destination device in the same particular overlay network.
- Embodiments are directed to a system with one or more devices that include a hardware processor and that are configured to perform any of the operations described herein and/or recited in any of the claims below.
- a non-transitory computer readable storage medium comprises instructions which, when executed by one or more hardware processors, causes performance of any of the operations described herein and/or recited in any of the claims.
- the techniques described herein are implemented by one or more special-purpose computing devices.
- the special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination.
- ASICs application-specific integrated circuits
- FPGAs field programmable gate arrays
- NPUs network processing units
- Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques.
- the special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
- Figure 4 is a block diagram that illustrates a computer system 400 upon which an embodiment of the invention may be implemented.
- Computer system 400 includes a bus 402 or other communication mechanism for communicating information, and a hardware processor 404 coupled with bus 402 for processing information.
- Hardware processor 404 may be, for example, a general purpose microprocessor.
- Computer system 400 also includes a main memory 406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 402 for storing information and instructions to be executed by processor 404.
- Main memory 406 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 404.
- Such instructions when stored in non-transitory storage media accessible to processor 404, render computer system 400 into a special-purpose machine that is customized to perform the operations specified in the instructions.
- Computer system 400 further includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404.
- ROM read only memory
- a storage device 410 such as a magnetic disk or optical disk, is provided and coupled to bus 402 for storing information and instructions.
- Computer system 400 may be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT), for displaying information to a computer user.
- a display 412 such as a cathode ray tube (CRT)
- An input device 414 is coupled to bus 402 for communicating information and command selections to processor 404.
- cursor control 416 is Another type of user input device
- cursor control 416 such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 404 and for controlling cursor movement on display 412.
- This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
- Computer system 400 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 400 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions included in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequences of instructions included in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
- Non-volatile media includes, for example, optical or magnetic disks, such as storage device 410.
- Volatile media includes dynamic memory, such as main memory 406.
- Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, content-addressable memory (CAM), and ternary content- addressable memory (TCAM).
- a floppy disk a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium
- CD-ROM any other optical data storage medium
- any physical medium with patterns of holes a RAM, a PROM, and EPROM
- FLASH-EPROM any other memory chip or cartridge
- CAM content-addressable memory
- TCAM ternary content- addressable memory
- Storage media is distinct from but may be used in conjunction with transmission media.
- Transmission media participates in transferring information between storage media.
- transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 402.
- transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
- Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 404 for execution.
- the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer.
- the remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem.
- a modem local to computer system 400 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal.
- An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 402.
- Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions.
- Computer system 400 also includes a communication interface 418 coupled to bus 402.
- Communication interface 418 provides a two-way data communication coupling to a network link 420 that is connected to a local network 422.
- communication interface 418 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line.
- ISDN integrated services digital network
- communication interface 418 may be a local area network (FAN) card to provide a data communication connection to a compatible FAN.
- FAN local area network
- Wireless links may also be implemented.
- communication interface 418 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
- Network link 420 typically provides data communication through one or more networks to other data devices.
- network link 420 may provide a connection through local network 422 to a host computer 424 or to data equipment operated by an Internet Service Provider (ISP) 426.
- ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the “Internet” 428.
- Internet 428 uses electrical, electromagnetic or optical signals that carry digital data streams.
- the signals through the various networks and the signals on network link 420 and through communication interface 418, which carry the digital data to and from computer system 400, are example forms of transmission media.
- Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and communication interface 418.
- a server 430 might transmit a requested code for an application program through Internet 428, ISP 426, local network 422 and communication interface 418.
- the received code may be executed by processor 404 as it is received, and/or stored in storage device 410, or other non-volatile storage for later execution.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Pure & Applied Mathematics (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Probability & Statistics with Applications (AREA)
- Algebra (AREA)
- Evolutionary Biology (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Medical Informatics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Operations Research (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020237041602A KR20240004875A (ko) | 2021-05-03 | 2022-04-13 | 사용자 내비게이션 추천들에 대한 인간 판독가능 설명들의 구성 |
| EP22721207.3A EP4334845A1 (en) | 2021-05-03 | 2022-04-13 | Composing human-readable explanations for user navigational recommendations |
| CN202280032906.4A CN117256001A (zh) | 2021-05-03 | 2022-04-13 | 为用户导航推荐撰写人类可读解释 |
| JP2023568069A JP7811953B2 (ja) | 2021-05-03 | 2022-04-13 | ユーザナビゲーション推奨のための人間可読説明の作成 |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/302,429 US11625446B2 (en) | 2021-05-03 | 2021-05-03 | Composing human-readable explanations for user navigational recommendations |
| US17/302,429 | 2021-05-03 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022235404A1 true WO2022235404A1 (en) | 2022-11-10 |
Family
ID=81580357
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/US2022/024687 Ceased WO2022235404A1 (en) | 2021-05-03 | 2022-04-13 | Composing human-readable explanations for user navigational recommendations |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US11625446B2 (enExample) |
| EP (1) | EP4334845A1 (enExample) |
| JP (1) | JP7811953B2 (enExample) |
| KR (1) | KR20240004875A (enExample) |
| CN (1) | CN117256001A (enExample) |
| WO (1) | WO2022235404A1 (enExample) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11625446B2 (en) * | 2021-05-03 | 2023-04-11 | Oracle International Corporation | Composing human-readable explanations for user navigational recommendations |
| US12120182B2 (en) * | 2021-07-07 | 2024-10-15 | Daily Rays Inc. | Systems and methods for modulating data objects to effect state changes |
| CN118824239B (zh) * | 2024-07-30 | 2026-03-20 | 科大讯飞股份有限公司 | 基于可解释推荐的车载交互方法及装置 |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021004228A1 (zh) * | 2019-07-08 | 2021-01-14 | 汉海信息技术(上海)有限公司 | 推荐理由的生成 |
| WO2021023249A1 (zh) * | 2019-08-06 | 2021-02-11 | 北京三快在线科技有限公司 | 推荐理由的生成 |
Family Cites Families (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6922680B2 (en) * | 2002-03-19 | 2005-07-26 | Koninklijke Philips Electronics N.V. | Method and apparatus for recommending an item of interest using a radial basis function to fuse a plurality of recommendation scores |
| US20050096950A1 (en) * | 2003-10-29 | 2005-05-05 | Caplan Scott M. | Method and apparatus for creating and evaluating strategies |
| US7512487B1 (en) * | 2006-11-02 | 2009-03-31 | Google Inc. | Adaptive and personalized navigation system |
| US8156059B2 (en) * | 2007-02-02 | 2012-04-10 | Dunning Ted E | Indicator-based recommendation system |
| US20120150626A1 (en) * | 2010-12-10 | 2012-06-14 | Zhang Ruofei Bruce | System and Method for Automated Recommendation of Advertisement Targeting Attributes |
| US20120172025A1 (en) * | 2010-12-29 | 2012-07-05 | Nokia Corporation | Method and apparatus for providing recommendations based on preloaded models |
| US8600926B2 (en) * | 2011-03-29 | 2013-12-03 | Manyworlds, Inc. | Integrated interest and expertise-based discovery system and method |
| US20120278268A1 (en) * | 2011-04-28 | 2012-11-01 | Nokia Corporation | Method and apparatus for extracting explicit profile representation through adaptive recommender system |
| US11195057B2 (en) * | 2014-03-18 | 2021-12-07 | Z Advanced Computing, Inc. | System and method for extremely efficient image and pattern recognition and artificial intelligence platform |
| US9460085B2 (en) * | 2013-12-09 | 2016-10-04 | International Business Machines Corporation | Testing and training a question-answering system |
| US20150206445A1 (en) * | 2014-01-17 | 2015-07-23 | Originator Inc. | Multi-sensory learning with feedback |
| US20160203221A1 (en) * | 2014-09-12 | 2016-07-14 | Lithium Technologies, Inc. | System and apparatus for an application agnostic user search engine |
| CN107818105B (zh) * | 2016-09-13 | 2021-04-09 | 腾讯科技(深圳)有限公司 | 应用程序的推荐方法及服务器 |
| US10909442B1 (en) * | 2017-03-30 | 2021-02-02 | Amazon Technologies, Inc. | Neural network-based artificial intelligence system for content-based recommendations using multi-perspective learned descriptors |
| CN110326005A (zh) * | 2017-12-25 | 2019-10-11 | 索尼公司 | 信息处理设备、信息处理方法及程序 |
| KR102066119B1 (ko) * | 2018-02-09 | 2020-02-17 | 제노플랜코리아 주식회사 | 다중 분석을 수행하기 위한 분주 매핑 장치 및 그 동작 방법 |
| US11429862B2 (en) * | 2018-03-20 | 2022-08-30 | Sri International | Dynamic adaptation of deep neural networks |
| US11455493B2 (en) * | 2018-05-16 | 2022-09-27 | International Business Machines Corporation | Explanations for artificial intelligence based recommendations |
| US11250347B2 (en) * | 2018-06-27 | 2022-02-15 | Microsoft Technology Licensing, Llc | Personalization enhanced recommendation models |
| US11435873B1 (en) * | 2018-12-12 | 2022-09-06 | Amazon Technologies, Inc. | User interface for identifying substitute items |
| US20200394534A1 (en) * | 2019-06-14 | 2020-12-17 | Sap Se | Multi task oriented recommendation system for benchmark improvement |
| US11361239B2 (en) * | 2019-11-07 | 2022-06-14 | Fmr Llc | Digital content classification and recommendation based upon artificial intelligence reinforcement learning |
| US12073451B2 (en) * | 2020-06-12 | 2024-08-27 | Online Mobile Services Joint Stock Company | Systems for e-commerce recommendations |
| US20220093088A1 (en) * | 2020-09-24 | 2022-03-24 | Apple Inc. | Contextual sentence embeddings for natural language processing applications |
| US11467857B2 (en) * | 2020-10-13 | 2022-10-11 | Adobe Inc. | Intelligently sensing digital user context to generate recommendations across client device applications |
| US20220207352A1 (en) * | 2020-12-30 | 2022-06-30 | Capital One Services, Llc | Methods and systems for generating recommendations for counterfactual explanations of computer alerts that are automatically detected by a machine learning algorithm |
| US11625446B2 (en) * | 2021-05-03 | 2023-04-11 | Oracle International Corporation | Composing human-readable explanations for user navigational recommendations |
| US20220366295A1 (en) * | 2021-05-13 | 2022-11-17 | Intuit Inc. | Pre-search content recommendations |
-
2021
- 2021-05-03 US US17/302,429 patent/US11625446B2/en active Active
-
2022
- 2022-04-13 CN CN202280032906.4A patent/CN117256001A/zh active Pending
- 2022-04-13 KR KR1020237041602A patent/KR20240004875A/ko active Pending
- 2022-04-13 WO PCT/US2022/024687 patent/WO2022235404A1/en not_active Ceased
- 2022-04-13 JP JP2023568069A patent/JP7811953B2/ja active Active
- 2022-04-13 EP EP22721207.3A patent/EP4334845A1/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021004228A1 (zh) * | 2019-07-08 | 2021-01-14 | 汉海信息技术(上海)有限公司 | 推荐理由的生成 |
| US20220147845A1 (en) * | 2019-07-08 | 2022-05-12 | Hanhai Information Technology (Shanghai) Co., Ltd. | Generation of recommendation reason |
| WO2021023249A1 (zh) * | 2019-08-06 | 2021-02-11 | 北京三快在线科技有限公司 | 推荐理由的生成 |
Non-Patent Citations (3)
| Title |
|---|
| LEI LI ET AL: "Personalized Transformer for Explainable Recommendation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 5 June 2021 (2021-06-05), XP081981681 * |
| LI LEI CSLEILI@COMP HKBU EDU HK ET AL: "Generate Neural Template Explanations for Recommendation", PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT, ACMPUB27, NEW YORK, NY, USA, 19 October 2020 (2020-10-19), pages 755 - 764, XP058708884, ISBN: 978-1-4503-6859-9, DOI: 10.1145/3340531.3411992 * |
| See also references of EP4334845A1 |
Also Published As
| Publication number | Publication date |
|---|---|
| JP7811953B2 (ja) | 2026-02-06 |
| CN117256001A (zh) | 2023-12-19 |
| US11625446B2 (en) | 2023-04-11 |
| US20220350846A1 (en) | 2022-11-03 |
| JP2024519722A (ja) | 2024-05-21 |
| KR20240004875A (ko) | 2024-01-11 |
| EP4334845A1 (en) | 2024-03-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11494559B2 (en) | Hybrid in-domain and out-of-domain document processing for non-vocabulary tokens of electronic documents | |
| US20220391595A1 (en) | User discussion environment interaction and curation via system-generated responses | |
| EP4248333A1 (en) | Intelligent query editor using neural network based machine learning | |
| US11775759B2 (en) | Systems and methods for training and evaluating machine learning models using generalized vocabulary tokens for document processing | |
| US11074307B2 (en) | Auto-location verification | |
| US11507747B2 (en) | Hybrid in-domain and out-of-domain document processing for non-vocabulary tokens of electronic documents | |
| US11625446B2 (en) | Composing human-readable explanations for user navigational recommendations | |
| US20220327373A1 (en) | Generating navigational target recommendations using parallel neural networks | |
| US11573812B2 (en) | Generating a next best action recommendation using a machine learning process | |
| US12373502B2 (en) | Generating and presenting search filter recommendations | |
| US11823667B2 (en) | Contextually-adaptive conversational interface | |
| US12547515B2 (en) | Run-time modification of data monitoring platform metrics | |
| US11983209B1 (en) | Partitioning documents for contextual search | |
| US20250315840A1 (en) | Applying A Machine Learning Model To Generate A Ranked List Of Candidate Actions For Addressing An Incident | |
| US12524611B2 (en) | Training graph neural network to identify key-value pairs in documents | |
| US20230351176A1 (en) | Machine-learning-guided issue resolution in data objects | |
| CN113806401A (zh) | 数据流处理 | |
| EP4591164A1 (en) | Machine-learning model & interface for planning, predicting, and implementing cloud resource systems | |
| US20230289559A1 (en) | Human-understandable insights for neural network predictions | |
| US20260064963A1 (en) | Generating Requests For Custom Script Generation | |
| US20260037234A1 (en) | Generating Software Code Using Large Language Models | |
| US12468716B2 (en) | Content-based operation selection | |
| US20250209348A1 (en) | Intelligent Virtual Assistant For Conversing With Multiple Users | |
| US20260023722A1 (en) | Overcoming Prompt Token Limitations Through Semantic Driven Dynamic Schema Integration For Enhanced Query Generation | |
| US20250173556A1 (en) | Relevance-Based Filtering Of Machine-Learning-Generated Descriptions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22721207 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2023568069 Country of ref document: JP |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 202280032906.4 Country of ref document: CN |
|
| ENP | Entry into the national phase |
Ref document number: 20237041602 Country of ref document: KR Kind code of ref document: A |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 1020237041602 Country of ref document: KR |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2022721207 Country of ref document: EP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| ENP | Entry into the national phase |
Ref document number: 2022721207 Country of ref document: EP Effective date: 20231204 |