EP4058888A1 - Dynamische lokalisierung und extraktion eines benutzerschnittstellenelementzustands in einer benutzerschnittstelle, die von einem ereignis einer anderen benutzerschnittstelle abhängt - Google Patents

Dynamische lokalisierung und extraktion eines benutzerschnittstellenelementzustands in einer benutzerschnittstelle, die von einem ereignis einer anderen benutzerschnittstelle abhängt

Info

Publication number
EP4058888A1
EP4058888A1 EP20811101.3A EP20811101A EP4058888A1 EP 4058888 A1 EP4058888 A1 EP 4058888A1 EP 20811101 A EP20811101 A EP 20811101A EP 4058888 A1 EP4058888 A1 EP 4058888A1
Authority
EP
European Patent Office
Prior art keywords
interface
user interface
value
candidate
integration code
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP20811101.3A
Other languages
English (en)
French (fr)
Inventor
Stefan Magureanu
Marcus NÄSLUND
Andrey Melentyev
Henrik Ståhl
Stanislav Dzhumaev
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Klarna Bank AB
Original Assignee
Klarna Bank AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/680,408 external-priority patent/US20210141498A1/en
Priority claimed from US16/680,403 external-priority patent/US11366645B2/en
Priority claimed from US16/680,392 external-priority patent/US11379092B2/en
Priority claimed from US16/680,406 external-priority patent/US11442749B2/en
Priority claimed from US16/680,396 external-priority patent/US11726752B2/en
Application filed by Klarna Bank AB filed Critical Klarna Bank AB
Publication of EP4058888A1 publication Critical patent/EP4058888A1/de
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces

Definitions

  • users may interact with various interfaces of service providers. Users may desire to perform various processes with regards to the services of the service providers. A user is often required to complete complicated and complex processes to interact with a service provider using various interfaces. Furthermore, if the user begins the interaction process and wishes to complete at a later date, the user often has to restart the process from the beginning. Still further, the user may desire to but be unable to share configured interfaces with other users. Providing users with the ability to streamline such processes presents a challenge, especially because the various service providers often have their own particular implementations of their interfaces involving large amounts of sub-processes and other complexities.
  • FIG. 1 illustrates an example of dynamic identification of user interface elements through unsupervised exploration in accordance with an embodiment
  • FIG. 2 illustrates an example of identifying user interface elements in a provided interface and generating integration code in accordance with an embodiment
  • FIG. 3 illustrates an example of identifying an element in one interface that is dependent upon another element in another interface in accordance with an embodiment
  • FIG. 4 is a flowchart that illustrates an example of identifying an element in one interface that is dependent upon another element in another interface in accordance with an embodiment
  • FIG. 5 is a flowchart that illustrates an example of identifying an element in one interface that is dependent upon another element in another interface in accordance with an embodiment
  • FIG. 6 illustrates an example of identifying an element in one interface that is dependent upon a configuration of another element in another interface in accordance with an embodiment
  • FIG. 7 is a flowchart that illustrates an example of identifying an element in one interface that is dependent upon a configuration of another element in another interface in accordance with an embodiment
  • FIG. 8 illustrates an example of identifying an element in an interface added from a different interface in accordance with an embodiment
  • FIG. 9 is a flowchart that illustrates an example of identifying an element in an interface added from a different interface in accordance with an embodiment
  • FIG. 10 illustrates an example of identifying values in one interface that are dependent upon an element in another interface, in accordance with an embodiment
  • FIG. 11 is a flowchart that illustrates an example of identifying values in one interface that are dependent upon an element in another interface in accordance with an embodiment
  • FIG. 12 illustrates an example of an interface on a client device in accordance with an embodiment
  • FIG. 13 illustrates a computing device that may be used in accordance with at least one embodiment.
  • Techniques and systems described herein relate to providing users of various services, which may interact with the users through various interfaces, with integration code that can model and/or streamline various processes and functionalities of the various interfaces. Users may interact with various services using the integration code to streamline the interactions and increase efficiency.
  • An integration code may provide various technological advantages, such as the ability for a user device to streamline interactions with service providers without having to burden the user with navigating potentially complex interfaces of the service providers.
  • user interface source code is obtained.
  • a particular object in the user interface source code is located by at least identifying a set of objects in the user interface source code, passing each of the set of objects as input to a machine learning classifier to determine a set of candidate interface obj ects for the particular object, and, for each candidate interface object of the set of candidate interface objects, simulating human interaction with the candidate interface object, detecting a result of simulating the human interaction, and modifying a ranking of the set of candidate interface objects based on the result to produce a modified set of candidate interface objects.
  • the particular object is determined, based on the ranking from the modified set of candidate interface objects.
  • a user interface is caused to be displayed on a client device.
  • the human interaction with the particular candidate interface object is simulated as a result of a selection made via the user interface.
  • the term “user interface” is contemplated to include a graphical user interface or other types of interface provided to a user for interaction with interface elements.
  • a set of first button object candidates in a first user interface is determined.
  • a set of second button object candidates in the first user interface is also determined, with the set of second button object candidates being associated with a respective set of values.
  • a set of candidate user interfaces is executed by simulating human interaction with the set of first button object candidates.
  • a set of candidate rankings is produced by at least, for each second button object of the set of second button object candidates storing a set of respective initial states of the set of candidate user interfaces, simulating the human interaction with the second button object, refreshing the set of candidate user interfaces, and, for each candidate user interface of the set of candidate user interfaces, on a condition that a difference between a respective initial state of the candidate user interface and a current state of the candidate user interface has a correlation to a respective value associated with the second button object, increasing candidate rankings of the second button object and the candidate user interface.
  • a principal second button object of the set of second button object candidates and a principal user interface of the set of candidate user interfaces are designated based on the set of candidate rankings.
  • integration code is generated based on metadata of the principal second button, the principal user interface, and a corresponding first button object that executed the principal user interface.
  • the integration code is provided to a client device to cause, as a result of execution of the integration code by the client device, the client device to execute the first user interface, simulate the human interaction with the principal second button object, and simulate the human interaction with the corresponding first button object to execute the principal user interface.
  • a first user interface having a first interface object, a second interface object, and a third interface object is executed, with the third interface object being associated with a first option element and a second option element.
  • the third interface object is set to the first option element.
  • human interaction is simulated with the second interface object a first time.
  • the second user interface is executed as a result of simulating the human interaction with the first interface object, with the second user interface being in a first state that includes a first listed selection corresponding to the first option element. Further in the example, the third interface object is set to the second option element. In the example, the human interaction is simulated with the second interface object a second time.
  • the second user interface is refreshed to produce the second user interface in a second state.
  • integration code is generated as a result of verifying that a difference between the second state and the first state includes a second listed selection corresponding to the second option element.
  • one of the first option element or the second option element of the third interface object of the first user interfaces is selected a result of execution of the integration code by a client device.
  • a first user interface having a first button object and a second button object is executed, with the first button object being associated with a value, a label, and a digital image.
  • human interaction is simulated with the first button object.
  • a second user interface is executed as a result of simulating the human interaction with the second button object.
  • the second user interface is identified to include an occurrence of the label.
  • integration code is generated based on locations of the label and the digital image in the first user interface and the occurrence of the value in the second user interface.
  • a client device is caused to display data obtained based on the locations as a result of execution of the integration code by the client device.
  • a first user interface having a first button object and a second button object is executed, with the first button object being associated with a first value.
  • human interaction is simulated with the first button object a first time.
  • the human interaction is simulated with the second button object to execute a second user interface in a first state.
  • the human interaction is simulated with the first button object a second time. Further in the example, the second user interface is refreshed to produce the second user interface in a second state. Still in the example, a difference between a second value in the second user interface in the second state and another second value in the second user interface in the first state that has a numeric amount of one is identified.
  • a third value in the second user interface that is associated with the second value is identified based on the first value. Further in the example, locations of the second value and the third value in the second user interface are obtained. Still in the example, integration code is generated based on metadata that indicates the locations. Finally, in the example, the integration code is provided to a client device to cause, as a result of execution of the integration code by the client device, the client device to obtain the data from the locations in at least one user interface, and display the data in an interface of the client device.
  • an above-mentioned service provider may be a library organization.
  • the library organization may be one of many library organizations that utilize one or more interfaces that users may interact with to select and borrow books.
  • a system of the present disclosure may analyze the interfaces of the library organization in order generate integration code capable of, when executed, dynamically interacting with the interfaces for the benefit of the user.
  • the system may analyze the interfaces to classify various elements of the interfaces using one or more classification heuristics and/or machine learning algorithms, such as a recurrent neural network (RNN), convolutional neural network (CNN), a random forest classifier, keyword searches, regular expressions, digital image recognition algorithms, and/or variations or combinations thereof, to determine the identity and functionality of the various elements such that the integration code may incorporate and utilize the various elements.
  • the integration code may be code that models or executes various processes using the interfaces of the library organization.
  • the integration code may comprise multiple codes for executing various processes through the interfaces of the library organization, such as borrowing a book, returning a book, changing accounts, and the like.
  • the user may use the appropriate integration code (e.g., via a mobile device such as a smartphone) to select and borrow a book through the library organization.
  • the integration code may perform one or more processes using the interfaces of the library organization such that the selected book is automatically borrowed for the user.
  • a service provider may be a utility provider.
  • the utility provider may utilize one or more interfaces that users may interact with to perform various processes regarding utilities the users may utilize.
  • a user may desire to change an address registered with the utility provider.
  • the user may cause a system to analyze the interfaces of the utility provider and generate integration code.
  • the system may have already analyzed the utility provider and generated integration code.
  • the integration code may be code that models or executes various processes using the interfaces of the utility provider.
  • the integration code may comprise multiple codes for different processes, such as changing an address, updating utility options, activating/de-activating specific services, and the like.
  • the user may use the appropriate integration code to change the address registered with the utility provider.
  • the integration code may perform one or more processes using the interfaces of the utility provider such that the registered address is changed for the user.
  • the present disclosure provides various examples for use of the techniques described, such as for use by a library organization, for changing an address at a utility provider, for reserving a seat in a movie theater, and so on. These examples are provided for illustrative purposes, and it is contemplated that there are many different types of applications for these techniques.
  • Other example applications of the techniques described in the present disclosure include but are not limited to analyzing interfaces and generating integration code executable to perform automatic form filling (e.g., for a parent teacher association gathering), setting preferences in an account profile, changing cookie preferences for a website, performing website interaction for a sight-impaired user, and so on.
  • Techniques described and suggested in the present disclosure improve the field of computing, especially the field of computer interface interaction, by creating integration code that, when executed, allows a device to automatically configure elements of a user interface in accordance with a user’s preferences without requiring input from the user. Additionally, techniques described and suggested in the present disclosure improve the usability of computer systems by allowing users to share configured interfaces with other users even when the interface itself lacks the ability to do so. Moreover, techniques described and suggested in the present disclosure are necessarily rooted in computer technology in order to overcome problems specifically arising with being able to determine the functionality, values, and attributes of various elements of an interface by dynamically interacting with the elements and observing the results of the interaction.
  • FIG. 1 illustrates an example 100 of dynamic identification of user interface elements through unsupervised exploration in accordance with an embodiment.
  • the example 100 may include an integration code generator 102, which may interact via a simulated human interaction 126 with a first interface 104 through a set of first elements 120.
  • the integration code generator 102 may utilize the first interface 104 to interact with a set of second interfaces 112, which may comprise a set of second elements 122.
  • an interface element may refer to an individual component of the interface having one or more attributes.
  • an interface element may alternatively be referred to as an “object” of a document object model (DOM) of the interface.
  • DOM document object model
  • the integration code generator 102 may be any entity operable to access various systems and/or services.
  • the integration code generator 102 may be implemented as software, hardware, and/or variations thereof.
  • the integration code generator 102 may interact with various interfaces to identify the various interfaces as well as elements within the various interfaces.
  • the integration code generator 102 may be implemented as an application or service executing on a computing device, such as the computing device 1300 of FIG. 13.
  • Examples of such a computing device include one or more instances of a physical computing instance (e.g., a physical server computer, a mobile communication device, a laptop computer, a tablet computer, a personal computer, a mainframe, etc.) or one or more instances of a virtual computing instance, such as a virtual machine hosted on one or more computer servers, or other computing system capable of communicating with various systems and/or services.
  • the integration code generator 102 may interact with various interfaces, such as Internet webpages (e.g., markup language interfaces), computing applications, computing services, mobile devices, and/or variations thereof.
  • the simulated human interaction 126 is an element that the integration code generator 102 utilizes to interact with various interfaces.
  • the simulated human interaction 126 may vary in implementation depending on which interface the integration code generator 102 is interacting with.
  • the integration code generator 102 may be interacting with an Internet website through a user interface typically generated from source code written in Hypertext Markup Language or other comparable markup language (referred to as a “markup language interface”), in which the simulated human interaction 126 may be implemented by simulating (e.g., via execution of event handlers, such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.) a pointer or mouse cursor.
  • event handlers such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.
  • the integration code generator 102 may interact via a command-line interface, in which the simulated human interaction 126 may be implemented according to various text commands.
  • the simulated human interaction 126 is any element/entity that enables the integration code generator 102 to interact with a desired interface.
  • the first interface 104 may be an interface of a computing service provided by a service provider. Examples of services a computing service may provide include data processing, data storage, applications, security, encryption, and/or other such services.
  • the first interface 104 may be an interface of a service accessible through the Internet, such as a library service, utility service, television service, entertainment service, and/or variations thereof.
  • the first interface 104 may comprise the set of first elements 120, through which an entity may interact with the first interface 104.
  • the integration code generator 102 may analyze the first interface 104 as well as the set of first elements 120 in order to classify and/or identify the set of first elements. For example, the integration code generator 102 may analyze various elements of the first interface 104 (e.g., such as the source/software code of the first interface 104, the DOM of the first interface 104, and/or intercepted communications between a user device and the computing service) to determine information about the first interface 104. The integration code generator 102 may analyze the structure of the first interface 104 (e.g., a DOM of the interface) to identify a set of interface element nodes.
  • various elements of the first interface 104 e.g., such as the source/software code of the first interface 104, the DOM of the first interface 104, and/or intercepted communications between a user device and the computing service
  • the integration code generator 102 may analyze the structure of the first interface 104 (e.g., a DOM of the interface) to identify a set of interface element nodes.
  • the integration code generator 102 may pass the set of interface element nodes as input to one or more machine learning classifiers and/or heuristics in order to classify the set of interface element nodes.
  • the resulting set of classified interface element nodes (also referred to as a set of classified interface objects) may be organized/ranked in accordance with a likelihood of being a certain type of interface object. For example, if an interface object having the object type of “button” is sought, the set of classified interface element nodes may be listed in order of likelihood that they are “button” objects. Likewise, if an interface object having the object type of “radio button” is sought, the set of classified interface element nodes may be listed in order of likelihood that they are “radio button” objects.
  • buttons or radio buttons refer to graphical control elements in a user interface that can be interacted with.
  • interface control objects include, but are not limited to, buttons, radio buttons, checkboxes, drop-down lists, text/number fields, hidden fields, date/time controls, telephone number fields, Uniform Resource Locator fields, color pickers, email address fields, range controls, file upload controls, search fields, and other such control objects. It is further contemplated that such interface control objects need not be limited to Hypertext Markup Language control objects, but may include option selection controls in other types of interfaces having similar characteristics.
  • buttons objects may refer to any graphical control element that enables an entity to trigger an event by interacting with the graphical control element.
  • a “radio button” may refer to any graphical control element that allows an entity to choose only one of a predefined set of mutually exclusive options.
  • a “checkbox” may refer to any graphical control element that allows an entity to make a binary choice between two mutually exclusive options.
  • a “drop-down list” may refer to any graphical control element that allows an entity to choose one value from a list of values.
  • the integration code generator 102 may determine a subset of the set of classified interface element nodes to be the set of first elements 120. The integration code generator 102 may further determine which of the set of first elements 120 perform the function(s) sought in a manner as described below.
  • the first interface 104 may be a markup language interface of a library organization that enables its patrons to search for and reserve books and/or to take advantage of other services provided by the library, and the integration code generator 102 may analyze the set of first elements 120.
  • the analysis may involve the integration code generator 102 selecting various elements of the set of first elements 120 via the simulated human interaction 126, and analyzing the results and/or corresponding interfaces that may open in response to the selections.
  • the integration code generator 102 may select each element of the set of first elements 120, analyze the corresponding results (e.g., the effects of selecting one of the first elements in the interface), and determine that the set of first elements 120 are elements that perform the functionality sought (e.g., adding books to be borrowed to a queue, viewing currently borrowed books, selecting chapters of books to be viewed online or copied, etc.).
  • the set of second interfaces 112 may be a set of interfaces resulting from interaction with the first interface 104. In some examples, the set of second interfaces 112 may be generated in response to the integration code generator 102 interacting with the set of first elements 120 of the first interface 104 via the simulated human interaction 126.
  • the integration code generator 102 may interact with the set of first elements 120 and analyze the corresponding set of second interfaces 112 to determine the identities of the set of second interfaces 112.
  • the integration code generator 102 may select an element of the set of first elements 120.
  • an interface of the set of second interfaces 112 may open in response to the element selection and may be an interface that displays currently borrowed books.
  • the integration code generator 102 may analyze the opened interface and determine that it is displaying currently borrowed books.
  • the integration code generator 102 may continue this process until the set of second interfaces 112 and/or set of second elements 122 within the set of second interfaces is correctly identified.
  • the set of second interfaces 112 may be interacted with through the set of second elements 122.
  • the set of second elements 122 may be interacted with by the integration code generator 102 via the simulated human interaction 126 to form a set of third interfaces in a same or similar manner that the set of second interfaces 112 were formed from interaction with the set of first elements 120 of the first interface 104 by the integration code generator 102 via the simulated human interaction 126.
  • the integration code generator 102 may continuously select various elements, and analyze corresponding interfaces that open, to identify all elements and resulting interfaces from the given interface.
  • the integration code generator 102 may utilize various classification heuristics and/or machine learning algorithms.
  • FIG. 2 illustrates an example 200 of identifying user interface elements in a provided interface and generating integration code in accordance with an embodiment.
  • the example 200 includes an interface provider 204, which may provide an initial interface 206 via a network 208 to an integration code generator 202, which may determine a DOM 210 corresponding to the initial interface 206 and a set of candidate interfaces 212 that are executed in response to engaging (e.g., simulating human interaction with) interface controls in the initial interface 206.
  • the integration code generator 202 may generate, based on the information gathered from interacting with the initial interface 206 and the set of candidate interfaces 212, integration code 214, which may be provided to a client device 216 operated by a user 218.
  • the integration code generator 202 may be an entity similar to the integration code generator 102 as described in connection with FIG. 1.
  • the integration code generator 202 may interact with the interface provider 204 through the network 208.
  • the network 208 may be any path of communication between the interface provider 204, integration code generator 202, and client device 216.
  • the network 208 may include the Internet, a local area network, a wide area network, Wi-Fi, a cellular network, and/or variations thereof.
  • the interface provider 204 may be a computing service that may provide interfaces to interact with its various services as described in the present disclosure.
  • entities may interact with the interface provider 204 through the initial interface 206.
  • the interface provider 204 may be operated by a library organization.
  • the initial interface 206 may be a home markup language interface of the library organization, through which entities may interact with to access services of the library organization.
  • the integration code generator 202 may receive the initial interface 206 through the network 208.
  • the integration code generator 202 may select various user interface elements of the initial interface 206 to produce the set of candidate interfaces 212.
  • the integration code generator 202 may analyze the set of candidate interfaces 212, as well as various elements of the initial interface 206, to determine the DOM 210.
  • the DOM 210 is a hierarchical model in a tree structure whereby each node is an object representing part of the interface.
  • the DOM 210 may comprise a model of the interface objects of the interfaces the interface provider 204 provides, such that the model may be utilized to determine classifications for the objects within the model.
  • the integration code generator 202 may utilize various heuristics, machine learning, keyword searches, regular expressions, digital image recognition, and/or artificial intelligence classification algorithms to classify the objects in the DOM 210.
  • the classification method may additionally involve a distance function. For example, a keyword search may find not only exact matches but may find and/or rank inexact matches based on closeness to the keyword; for example a keyword search for the search string “ha” may also return “hat,” noting that it has a distance of one because one letter is not an exact match to the search string.
  • certain functionalities of the initial interface 206 and/or the set of candidate interfaces 212 objects may be identified.
  • the initial interface 206 may be a home markup language interface of a library organization.
  • the integration code generator 202 may receive the initial interface 206, and select (e.g., by simulating human interaction with) various elements of the initial interface 206 to produce the set of candidate interfaces 212.
  • the set of candidate interfaces 212 may themselves be markup language interfaces that are executed as a result of engaging with elements (such as button objects or other interface control objects) of the initial interfaces; the set of candidate interfaces may, for example, be produced by simulating selection of a link to a different markup language interface or may execute in a separate window as a result of simulating a selection of a button element.
  • the set of candidate interfaces 212 may be interfaces displaying information such as account information, currently borrowed books, outstanding fines, and various other metrics and services the library organization may utilize and/or provide, and may be accessible through the initial interface 206.
  • the integration code generator 202 may analyze the candidate interfaces 212 to determine the identities of the various interfaces; in some examples, the analysis may comprise determining which candidate interfaces 212 correspond to which elements of the initial interface 206.
  • the integration code generator 202 may analyze the DOM 210 that models the initial interface 206 and the set of candidate interfaces 212.
  • the DOM 210 may include information regarding the functions/identities of each element of the initial interface 206, as well as the identities of the set of candidate interfaces 212 and the functionalities that may be associated with the interfaces.
  • the integration code 214 may be executable code configured to interact with the interface provider 204 through one or more interfaces, such as the initial interface 206 and/or set of candidate interfaces 212.
  • the integration code 214 may include code to execute a specific process.
  • the integration code 214 may include code that causes the selection of one or more elements of an interface to accomplish a specified task.
  • the integration code 214 may include code that identifies elements and interfaces such that an entity may utilize the elements and interfaces appropriately.
  • the integration code 214 may be transmitted to the client device 216, whereupon the user 218 may utilize the integration code 214 to interact with the interface provider 204 via the client device 216.
  • the user 218 may be any entity operable to access various systems and/or services such as the interface provider 204.
  • the user 218 may interact with various systems and/or services through the client device 216, which may be any operable device, such as a mobile device, computer, and/or variations thereof.
  • the user 218 may desire to interact with the interface provider 204, which may be a library organization. Continuing with this example, the user 218 may desire to borrow a book from the library organization through the initial interface 206.
  • the user 218 may cause the integration code generator 202 to analyze various elements and interfaces of the library organization, and determine the appropriate integration code 214 to borrow a book utilizing the various elements and interfaces of the library organization.
  • the user 218 may receive the integration code 214 via the client device 216, and execute the integration code 214 on the client device 216 such that the desired task for the user 218 is accomplished.
  • the integration code 214 may be executed by a client device, a remote server, and/or variations thereof, to perform any process in connection with the interface provider 204.
  • the integration code is in a programming language that can be executed by a device with one or more processors.
  • the integration code 214 may not be strictly executable itself, but may be structured data (e.g., a configuration file, key/value pairs, etc.) that may be processed by a software application running on such a device (e.g., the client device 216) to enable the software application to interact via the device with the interface elements of the interfaces provided by the interface provider 204.
  • this alternate embodiment is also contemplated.
  • the operations of FIG. 2 may be utilized in any order, including parallel.
  • the integration code 214 may be generated in real-time as the user 218 interacts with the interface provider 204 via the client device 216.
  • the integration code 214 may be transmitted to the user 218 as the user 218 is interacting with the interface provider 204; the user 218 may then utilize the integration code 214 to streamline interactions with the interface provider 204.
  • the integration code generator 202 may preemptively analyze and determine integration code for a variety of computing services/providers, and provide the integration codes for users desiring certain functionalities from the computing services/providers.
  • FIG. 3 illustrates an example 300 of identifying an element in one interface that is dependent upon another element in another interface in accordance with an embodiment.
  • the example 300 includes a first interface at a first time 304 A and an initial second interface 312A, which may be associated with each other.
  • the initial second interface 312A may also include interface elements at the first time 320A.
  • the example further includes the first interface at a second time 304B and the refreshed second interface 312B, which may be associated with each other.
  • the refreshed second interface 312B may also include interface elements at the second time 320B.
  • a simulated human interaction 326 may be utilized to interact with the interfaces.
  • the first interface at a first time 304A may be an interface of an interface provider, which may provide various services as described in the present disclosure.
  • the first interface at the first time 304 A may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the initial second interface 312A may be a different interface of the same interface provider that provides the first interface at a first time 304 A.
  • the initial second interface 312A may be similar to one of the set of second interfaces 112 of FIG. 1 or one of the set of candidate interfaces 212 of FIG. 2.
  • the second interface may be related to the first interface such that interactions with elements in the first interface may cause changes to the elements in the second interface (upon refresh).
  • the interface provider may be a library organization that provides library services.
  • the first interface at a first time 304A may be an interface displaying selectable potential books to borrow
  • the initial second interface 312A may be an interface displaying a total of selected potential books.
  • the interface elements at the first time 320A may be elements of the initial second interface 312A denoting various actions/values.
  • the interface elements at the first time 320A may be associated with various elements of the first interface at a first time 304 A.
  • elements of the first interface at a first time 304A may be associated with elements of the initial second interface 312 A, and vice versa.
  • interface elements at the first time 320A in the initial second interface 312A includes six elements depicting values.
  • the “Value C,” as can be seen is equal to some other value, “B ” Values “D” and ⁇ ” are depicted in the initial second interface 312A, and the “Value F” can be seen to be a summation of “C” plus “D” with a reduction of an amount ⁇ ”
  • the “Value H” can be seen to be equal to a value “G” multiplied against “F.”
  • the “Value I” can be seen to be a summation of “F” and “H.”
  • the selection of an element on the first interface at a first time 304 A may affect a value of the interface elements at the first time 320A, as displayed in the refreshed second interface 312B.
  • the “Value C” is now equal to the previous value “B” plus the amount of “A” from the first interface at the first time 304A.
  • the values of “F,” “H,” and “I,” being dependent upon the value of “C,” may likewise change according to their mathematical correspondence.
  • the integration code generator 102 may attempt to identify an element of the initial second interface 312A that is dependent on an element of the first interface at a first time 304 A.
  • the integration code generator may perform one or more processes to identify the element of the initial second interface 312A that is dependent on an element of the first interface at a first time 304 A.
  • the integration code generator may perform one or more processes to obtain the first interface at a first time 304 A and the initial second interface 312A.
  • the integration code generator may be provided with the first interface at a first time 304A and the initial second interface 312A. In some examples, the integration code generator may utilize an input from the entity to obtain the first interface at a first time 304A and the initial second interface 312A. Additionally, in some examples, the integration code generator may be provided with the first interface at a first time 304 A, and interact with one or more elements of the first interface at a first time 304 A to obtain the initial second interface 312A.
  • the integration code generator may save a current state (also referred to as a “present state”) of the first interface at the first time 304 A and, likewise, a current state of the initial second interface 312A.
  • the first interface at a first time 304A and the initial second interface 312 A may be Internet markup language interfaces, in which a Hypertext Markup Language (HTML) representation of the first interface at a first time 304 A and the initial second interface 312A may be stored.
  • HTML Hypertext Markup Language
  • a representation of both the first interface at a first time 304A and the initial second interface 312A may be stored by the integration code generator.
  • the integration code generator may analyze the elements of the first interface at a first time 304A.
  • the integration code generator may utilize a database comprising various known elements and definitions based upon other conventional interfaces.
  • the integration code generator may utilize various keyword searches, regular expressions, digital image recognition algorithms, neural networks, and/or machine learning algorithms to detect and identify elements of the first interface at a first time 304A (e.g., by passing a set of interface element nodes identified as input to one or more machine learning classifiers in order to classify interface element nodes).
  • the integration code generator may determine if any elements of the first interface at a first time 304 A match any potentially known elements. In various embodiments, a match does not necessarily require equality.
  • two values may match if they are not equal but are mathematically equivalent.
  • two values may match if they correspond to a common object (e.g., value) or are in some predetermined way complementary and/or they satisfy one or more matching criteria.
  • a common object e.g., value
  • the integration code generator may determine a classification and functionality for the element.
  • the integration code generator may additionally analyze the interface elements at the first time 320A.
  • the integration code generator may utilize the database as described above, as well as various other algorithms, to attempt to classify the interface elements at the first time 320A.
  • the integration code generator may assign a value, or candidate rank, to each interface element of the interface elements at the first time 320A, in which the value may denote a level of confidence that the integration code generator has in determining the identity of a particular element.
  • the candidate rank value may be assigned by, for each of the interface elements, passing attributes of the interface elements as input to a set of machine learning classifiers that has been trained to classify interface elements in a DOM.
  • the integration code generator may pass the source path of the digital image to a digital image recognition algorithm to attempt to determine what the digital image depicts; this determination may, in turn, be passed to another machine learning classifier algorithm to determine the classification of an interface element associated with such a digital image. For example, a digital image classification algorithm may return that a digital image associated with an interface element is a symbol for a printer, and a next machine learning classifier algorithm may classify the interface element as a “Send to Printer” type control object.
  • the integration code generator may assign all elements a default value and update the value upon further processing/analysis. It is also contemplated that where the present disclosure discusses increasing or decreasing a rank value that, in some embodiments, the rank value may be binary such that the rank value is either true or false.
  • the integration code generator may determine, as result of passing the attributes of the element “Button 1” through the set of machine learning classifiers, that the element “Button 1” is an element of the first interface at a first time 304 A that can be selected/engaged.
  • the element “Button 1” may be a button object or other interface control.
  • the integration code generator may additionally determine, via the set of machine learning classifiers, the functionality of the element “Button 1” based upon historical usages of similar elements.
  • the integration code generator may additionally determine that the element “Button 1” may be associated with the element “Value A.”
  • the set of machine learning classifiers may provide a rank value that indicates, based on factors such as proximity to the element “Button 1,” that the element depicting “Value A” is associated with the element “Button 1.”
  • the integration code generator may then utilize the simulated human interaction 326, at a second time, to interact with the first interface at a second time 304B.
  • the first interface at the second time 304B may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the simulated human interaction 326 may be the result of an entity or execution of a function that enables the integration code generator to interact with a desired interface.
  • the simulated human interaction 326 may be implemented as triggering a button or touch event (e.g., onClick, onMouseDown, onMouseUp, etc.).
  • the integration code generator may simulate human interaction by simulating an input device (e.g., mouse, trackball, keyboard, etc.) to manipulate a pointer object (e.g., mouse cursor) to overlay the interface element and then simulate one or more “clicks” or other event to engage the interface element.
  • an input device e.g., mouse, trackball, keyboard, etc.
  • a pointer object e.g., mouse cursor
  • the integration code generator may utilize the simulated human interaction 326 to select the selectable element “Button 1.”
  • the integration code generator may then obtain the refreshed second interface 312B.
  • the refreshed second interface 312B may be obtained by refreshing, reloading, or otherwise updating the second interface.
  • the initial second interface 312A may be an Internet markup language interface, in which the refreshed second interface 312B may be obtained by refreshing the markup language interface or retrieving the markup language interface again.
  • the integration code generator may then compare the interface elements at the second time 320B with the interface elements at the first time 320A, which may be stored in various representations by the integration code generator.
  • the refreshed second interface 312B may not be a separate interface from the first interface at the second time 304B, but may instead be a document object model mutation to the first interface at the second time 304B (e.g., an overlay that appears when a set of steps are performed, such as when a cursor hovers over or “clicks” an element in the first interface at the second time 304B).
  • a document object model mutation to the first interface at the second time 304B (e.g., an overlay that appears when a set of steps are performed, such as when a cursor hovers over or “clicks” an element in the first interface at the second time 304B).
  • the integration code generator is attempting to determine an element in the second interface that has a value that is related to, but most attenuated from, a value of an element in the first interface by comparing a state of the initial second interface 312A with a state of the refreshed second interface 312B.
  • the integration code generator upon analyzing the current state of the refreshed second interface 312, may determine that, upon the selection of the “Button 1” element, the value of the element depicting “Value C” is seen to have changed in response.
  • the integration code generator may determine that the element depicting “Value A” from the first interface has a relationship with the element depicting “Value C” in the second interface.
  • the integration code generator may then update the candidate ranks of the interface elements such that a candidate rank value for the element depicting “Value C” is increased to denote that the integration code generator has increased confidence in its determination of the identity and functionality of the element depicting “Value C.”
  • the integration code generator may further determine that because the value of the element depicting “Value C” has increased by the value indicated by the element depicting “Value A,” and further increase the candidate rank value for “Value A.”
  • the integration code generator may further recognize that “Value F,” “Value H,” and “Value I” have likewise changed in accordance with a relationship to “Value A.” That is, because the element depicting “Value C” has increased by the amount “A” in the refreshed second interface 312B, that the “Value F,” which is the sum of “C” plus “D” minus ⁇ ,” likewise increases by the amount “A.” Therefore, the integration code generator may further increase the candidate rank value for the element corresponding to “Value
  • the integration code generator may recognize that by comparing the “Value H” in the initial second interface 312A with the “Value H” in the refreshed second interface 312B, the “Value H” is equal to the “Value F” multiplied by a fixed amount “G.”
  • the fixed amount “G” may be a first percentage value if the client device is associated with or located in a first geographic region but may be a second percentage value if the client device is associated with or located in a second geographic region.
  • the value “D” may be a fixed amount that is based on a distance between a first location associated with the entity interacting with the first and second interfaces (e.g., a user, a client device of the user, the integration code generator) and a second location (e.g., a location associated with the interface provider).
  • a first location associated with the entity interacting with the first and second interfaces e.g., a user, a client device of the user, the integration code generator
  • a second location e.g., a location associated with the interface provider.
  • the greater the distance between the first location and the second location the greater the value “D” may be.
  • the value “D” may be a number of days a book may be checked out, which may be based on how far from the library the user resides.
  • the integration code generator may determine that the “Value H” has an increased likelihood of having an association with “Value A” from the first interface, and the integration code generator may increase the candidate rank value for the element depicting “Value H” as well. Furthermore, the integration code generator may recognize that “Value I” is equal to the sum of “Value F” and “Value H.” Consequently, because “Value H” is mathematically related to “Value F,” which, as noted above, is mathematically related to “Value A,” the integration code generator may likewise increase a candidate rank value for the element depicting “Value I.”
  • the integration code generator may correspondingly decrease the candidate ranks of the remaining interface elements, as the integration code generator may determine that the remaining interface elements were unaffected by the selection of the element “Button 1 ” In alternative embodiments, the integration code generator may leave the candidate ranks of the remaining interface elements unchanged.
  • the integration code generator may further determine, after analyzing the first and second interfaces at the first and second times, that the values “J” and “K,” depicted in the first interface, have no correlation to any of the values in the second interface. Consequently, the elements depicting the values “J” and “K” may be determined not to be elements related to a value of an element in the second interface, and may be omitted from further analysis.
  • the integration code generator may, based on the ranking and mathematical relationships, determine that the element depicting the “Value I” is the element related to, but most attenuated from, a value of the element depicting “Value A” in the first interface.
  • the integration code generator may further affect the ranking based on other criteria, such as by increasing the candidate rank value or weighting the candidate rank value of an element based on other attributes of the element, such as bold text, a higher or lower value (from other candidate elements), a symbol or keyword (e.g., “Total,” “Subtotal,” etc.) found in another element proximate to the element in question.
  • “Value I” is in bold text, which may cause its ranking by the integration code generator to be increased by a predetermined weight value.
  • “Value F” may be higher than “Value C,” which may result in the candidate ranks for the element depicting “Value F” to be more heavily weighted in favor of being the particular element sought.
  • “Value I” may be still higher from “Value F,” which may result in the candidate ranks for the element depicting “Value I” to be even more heavily weighted in favor of being the particular element sought.
  • determining a correlation between the “Value A” and a value, such as “Value I,” need not be so complex as in the illustrated example, but may be that “Value I” is an amount that is within a tolerance threshold of “Value A.”
  • a higher candidate rank value indicates a better match to a sought element
  • a lower candidate rank value may indicate a better match.
  • the candidate rank value may be binary such that the candidate element may be determined to either be the sought element or not.
  • the element depicting “Value I” may result in the highest candidate rank value and be determined to be the sought value. Based on this determination and metadata about the element depicting “Value I” (e.g., element identifier, location within the HTML source code, etc.), the integration code generator may generate integration code that, upon execution by a client device, causes the client device to locate the corresponding element in another version of the second interface provided by the interface provider, extract the target value of the element.
  • integration code generator may generate integration code that, upon execution by a client device, causes the client device to locate the corresponding element in another version of the second interface provided by the interface provider, extract the target value of the element.
  • the client device may, via an application running on the client device, may perform one or more operations utilizing the target value; for example, the client device may display the target value, may compare the target value against some other value (e.g., a threshold, a value associated with the individual associated with the client device, another value extracted from another interface, etc.), may store the target value (e.g., in a browser cookie, in a local or remote data store, in a user session, etc.), transmitting the target value to a third party entity, or perform some other operation.
  • some other value e.g., a threshold, a value associated with the individual associated with the client device, another value extracted from another interface, etc.
  • some other value e.g., a threshold, a value associated with the individual associated with the client device, another value extracted from another interface, etc.
  • may store the target value e.g., in a browser cookie, in a local or remote data store, in a user session, etc.
  • the example 300 depicted in FIG. 3 has been simplified to a single first interface and single second interface for ease of explanation. It is contemplated, however, that in embodiments the integration code generator may perform the analysis described above for multiple first and second interfaces of the same interface provider in order to arrive at a final determination of the relationships between elements in the first interface and elements in the second interface. In various embodiments, this process may be repeated to identify all elements of the first and second interfaces. In some examples, the integration code generator may repeatedly simulate human interaction with various elements of the first interface, detect changes in the second interface in response to the simulated human interactions, and determine the relationships, dependencies, functionalities, and/or identities of elements of the first and second interfaces.
  • Algorithm I Location and Extraction of a User Interface Element State in a User Interface that is Dependent on an Event Occurrence in a Different User Interface
  • Algorithm takes as input several first interfaces (at least 3). Start from one of these interfaces.
  • Refresh second interface Extract all interface elements that approximately match the particular element types again. Match these particular element values to the particular values on the snapshot of the interface at step 2). (In embodiments, this is done using XML Path Language (XPath) and Cascading Style Sheets (CSS) selectors of the elements). In embodiments, at this step, identify which elements were present on the second interface before engaging the first button, and which elements appeared afterwards.
  • XPath XML Path Language
  • CSS Cascading Style Sheets
  • step 5 For the elements that are “alive” and existed previously, mark the ones whose value changed. If the value of such an element changed by a number that is not in the set of “particular element candidates,” this element is given a “flag.” If the number of flags is greater than the “tolerance threshold” defined in the initialization step, this element is discarded from further consideration - (e.g., marked as “dead”). The rank value of all elements remaining “alive” is now increased by 1. a. Repeat from step 2.
  • FIG. 4 is a flowchart illustrating an example of a process 400 for identifying an element in one interface that is dependent upon another element in another interface in accordance with various embodiments. Some or all of the process 400 (or any other processes described, or variations and/or combinations of those processes) may be performed under the control of one or more computer systems configured with executable instructions and/or other data, and may be implemented as executable instructions executing collectively on one or more processors.
  • the executable instructions and/or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer software program persistently stored on magnetic, optical, or flash media).
  • a non-transitory computer-readable storage medium e.g., a computer software program persistently stored on magnetic, optical, or flash media.
  • process 400 may be performed by any suitable system, such as the computing device 1300 of FIG. 13.
  • the process 400 includes a series of operations wherein user interface source code of an initial interface is obtained, a pair of object observers are executed to produce a set of first candidate interface objects and a set of second candidate interface objects, and then, engaging with each of the first candidate interface objects to determine if a new user interface is executed as a result, and adding the user interfaces that are executed to a set of candidate user interfaces.
  • the candidate interface objects may be button objects (such as “submit” buttons) or other interface controls.
  • the system proceeds to perform the process 400 of FIG. 5.
  • the system performing the process 400 may desire to identify elements of various interfaces, which may be provided by an interface provider.
  • the system may be a system such as the integration code generator 102 as described in connection with FIG. 1.
  • the system may obtain user interface source code of an initial interface.
  • the system may obtain the source code by interfacing with the interface provider (e.g., establishing a network connection and receiving the source code from the interface provider via the network connection).
  • the system may be provided with (e.g., pushed) the user interface source code.
  • the system may execute a first object observer to produce a set of first candidate interface objects.
  • the first object observer may be an application, computer software program, or otherwise any entity that may analyze an interface on behalf of the system.
  • the first object observer may utilize various heuristics, algorithms, processes, and/or variations thereof, to detect and analyze various interface objects.
  • the first object observer may analyze the user interface/initial interface to determine the set of first candidate interface objects.
  • the set of first candidate interface objects may be elements of the user interface/initial interface that may be interactive.
  • the first object observer may be configured to detect candidate interface objects that may potentially cause the execution of a new user interface from the initial interface.
  • the first object observer may additionally predict various characteristics of the candidate interface objects of the set of first candidate interface objects, such as functionality, dependencies, and/or variations thereof.
  • the system may execute a second object observer to produce a set of second candidate interface objects.
  • the second object observer may be an entity similar to the first object observer.
  • the second object observer may be configured to detect candidate interface objects that may affect elements in other interfaces.
  • the second object observer may also predict which specific elements are affected by which candidate interface objects of the set of second candidate interface objects.
  • the system may assign candidate ranks to the objects of the set of second candidate interface objects.
  • each object may be assigned a default candidate rank.
  • the candidate rank value may represent a degree of confidence that the system has in determining the identity and functionality of a given object.
  • the system may engage a first/next object of the set of first candidate interface objects.
  • the system may engage the first/next object through the usage of simulated human interaction and/or variations thereof.
  • the system in 410, may determine if engaging the object executes a new user interface. If engaging the object does not execute a user interface, the system performing the process 400 may return to 408 to engage with the next first candidate interface object until engaging a candidate interface object of the set of first candidate interface objects results in the execution of a new user interface. If engaging the object executes a new user interface, the system may proceed to 412.
  • the system upon determining that engaging the object results in a new user interface, may add the new user interface to a set of candidate user interfaces.
  • the system may determine if the object engaged with is the last first candidate interface object. If the object engaged with is the last first candidate interface object, the system performing the process 400 may proceed to 416, whereupon the system may perform process 500 illustrated in FIG. 5.
  • the system may repeat the operations of 408-14 for all of the objects of the set of first candidate interface objects.
  • the system in 416, may proceed to 502.
  • the system may assign candidate ranks to the user interfaces of the set of candidate user interfaces.
  • each user interface may be assigned a default candidate rank.
  • the candidate rank value may represent a degree of confidence that the system has in determining the identity and functionality of a given user interface. Note that one or more of the operations performed in 402-16 may be performed in various orders and combinations, including in parallel.
  • FIG. 5 is a flowchart illustrating an example of a process 500 for identifying an element in one interface that is dependent upon another element in another interface in accordance with various embodiments.
  • Some or all of the process 500 may be performed under the control of one or more computer systems configured with executable instructions and/or other data, and may be implemented as executable instructions executing collectively on one or more processors.
  • the executable instructions and/or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer software program persistently stored on magnetic, optical, or flash media).
  • a non-transitory computer-readable storage medium e.g., a computer software program persistently stored on magnetic, optical, or flash media.
  • some or all of process 500 may be performed by any suitable system, such as the computing device 1300 of FIG. 13.
  • the process 500 includes a series of operations wherein each of the second candidate interface objects from the process 400 of FIG. 4 are engaged, each of the candidate user interfaces is refreshed, and if the candidate user interface changes as expected, and candidate ranks are increased for the candidate user interface, for the first candidate interface object that caused the candidate user interface to execute, and for the second candidate interface object that was engaged prior to the candidate user interface being refreshed.
  • the final (target) first interface object, the final (target) second interface object, and the final (target) user interface may be determined based on the aforementioned candidate ranks.
  • the candidate interface objects may be button objects (such as “submit” buttons) or other interface controls.
  • the system performing the process 500 may desire to identify elements of various interfaces, which may be provided by an interface provider.
  • the system may be a system such as the integration code generator 102 as described in connection with FIG. 1.
  • the process 500 is a continuation of the process 400.
  • the system may engage the first/next second candidate interface object of the set of second candidate interface objects. In some examples, the system may engage the first/next object through the usage of simulated human interaction and/or variations thereof.
  • the system may refresh first/next user interface of the set of candidate user interfaces. In some examples, the system may refresh a user interface by updating the user interface, which may be done by requesting a current version of the interface from the interface provider or through one or more processes from the system.
  • a user interface may be a markup language interface, which the system may refresh by reloading the markup language interface.
  • the system may determine if the candidate user interface changes as expected.
  • the system may expect various user interface changes based on predictions made from the determination and/or analysis of the set of second candidate interface objects. For example, the system may predict that a value depicted by an element in the candidate user interface will increase by an amount that exactly or approximately corresponds to a value depicted by an element in the initial interface. If the candidate user interface changes as expected, the system performing the process 500 may proceed to 510, whereupon the system may increase the candidate rank value of the candidate user interface. Then, in 512, the system may increase the candidate rank value of the first candidate interface object (that caused the candidate user interface to be executed) and/or second candidate interface object.
  • the system may omit increasing the candidate ranks of the candidate user interface and second interface object that was engaged with, and proceed to 514, whereupon the system may determine if the candidate user interface is the last user interface of the set of candidate user interfaces. If the candidate user interface is not the last user interface of the set of candidate user interfaces, the system may repeat the operations of 506-14 for all of the candidate user interfaces of the set of candidate user interfaces.
  • the system performing the process 500 may proceed to 516 to determine if the second candidate interface object that was engaged with is the last candidate interface object of the set of second candidate interface objects. If the second candidate interface object is not the last candidate interface object of the set of second candidate interface objects, the system may repeat the operations of 504-16 for each second candidate interface object of the set of second candidate interface objects. On the other hand, if the second candidate interface object is the last candidate interface object of the set of second candidate interface objects, the system may proceed to 518, whereupon the system may determine, based on the ranking of each, the final first interface object, the final second interface object, and the final user interface.
  • the final user interface is the candidate user interface of the set of candidate user interfaces with the highest candidate rank
  • the final second interface object is the second candidate interface object of the set of second candidate interface objects with the highest candidate rank
  • the final first interface object is a candidate interface object of the set of candidate interface objects, that, as a result of engaging the first interface object, results in the execution of the first user interface.
  • the system may determine that upon engaging the first interface object, the first user interface is executed in response, and, upon engaging the second interface object, one or more elements of the first user interface change in response to the engaging.
  • the system may further determine various characteristics of the dependency between the first user interface, the first interface object, and the second interface object.
  • an initial interface comprising the first interface object may be an interface of a library organization.
  • the first interface object may be a selectable element that allows an entity to view which books are currently borrowed, and may open the first user interface when selected, which may be an interface comprising the currently borrowed books.
  • the second interface object may be a selectable element denoting an action to borrow a specific book, in which, upon selection of the second interface object, the first user interface is updated to reflect the addition of the specific book as part of the currently borrowed books.
  • the operations performed in 502-18 may be performed in various orders and combinations, including in parallel.
  • the metadata e.g., object types, identifiers, attributes, locations in the interface source code, proximity to other elements, etc.
  • the integration code generator may use the integration code generator to generate integration code that, upon execution, causes a device to engage with corresponding interface objects and user interfaces of different first interfaces by the same interface provider.
  • the integration code generator may generate integration code that, if executed, allows a library book to be automatically added to a queue of a user regardless of the particular book or user. Note, however, that the processes 400 and 500 of FIGS.
  • the integration code generator has increased certainty that the identified interface objects and user interface are correct.
  • FIG. 6 illustrates such an interface where an option is selected prior to engaging a button object.
  • FIG. 6 illustrates an example 600 of identifying an element in one interface that is dependent upon a configuration of another element in another interface, in accordance with an embodiment.
  • the example 600 includes a first interface at a first time 604A comprising an option selection element at the first time 622A operable by a first simulated human interaction 626A, which may be associated with an initial second interface 612A comprising interface elements at the first time 620A (an initial time).
  • the option selection element may be any interface element from which one or more options may be selected.
  • Example option selection elements include but are not limited to drop-down list element objects, radio button element objects, and checkbox element objects.
  • the example 600 also includes a first interface at a second time 604B comprising an option selection element at the second time 622B operable by a second simulated human interaction 626B, which may be associated with a refreshed second interface 612B comprising interface elements at a second time 620B (subsequent to the initial time).
  • the first interface at a first time 604A may be an interface of an interface provider, which may provide various customizable services.
  • the first interface at the first time 604A and the first interface at the second time 604B may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the initial second interface 612A may be a different interface of the interface provider, and may denote various information originating from the first interface at a first time 604 A.
  • the initial second interface 612A may be similar to one of the set of second interfaces 112 of FIG. 1 or one of the set of candidate interfaces 212 of FIG. 2.
  • the second interface may be related to the first interface such that interactions with elements in the first interface may cause changes to the elements in the second interface (upon refresh).
  • the interface provider may be a cinema reservation service, in which the first interface at a first time 604A may be a seat customization interface, comprising options customizable through the option selection element at the first time 622A such as seat preference, heat preference, food preference, and/or variations thereof.
  • the initial second interface 612A may be an interface that, through the interface elements at the first time 620A, displays chosen options as an initial listed selection.
  • elements of the first interface may be associated with elements of the initial second interface 612A, and vice versa.
  • the selection of an option on the first interface may affect a value displayed on the initial second interface 612A and/or the initial second interface 612A.
  • the integration code generator 102 or similar entity may attempt to identify an element of the second interface that is dependent on an element of the first interface.
  • the integration code generator may perform one or more processes to obtain the first interface at a first time 604A.
  • the integration code generator may be provided with the first interface at a first time 604A and/or the initial second interface 612A.
  • the integration code generator may utilize an input from the entity to obtain the first interface at a first time 604A.
  • the integration code generator may analyze the first interface at a first time 604A.
  • the integration code generator may utilize various keyword searches, regular expressions, digital image recognition algorithms, neural networks, and/or machine learning algorithms to detect and identify elements of the first interface at a first time 604A (e.g., by passing a set of interface element nodes identified as input to one or more machine learning classifiers in order to classify interface element nodes).
  • the integration code generator may determine if any elements of the first interface at a first time 604A match any potentially known elements.
  • the integration code generator may determine that the selection of one or more elements of the first interface at a first time 604A results in the initial second interface 612A.
  • the integration code generator may additionally determine that the selectable element “Button 1” may affect the initial second interface 612A.
  • the element “Button 1” may be a button object (such as a “submit” button) or other interface control.
  • the integration code generator may select the element “Button 1” utilizing the first simulated human interaction 626A.
  • the integration code generator may also utilize the first simulated human interaction 626A to obtain the initial second interface 612A.
  • the first simulated human interaction 626A may be hardware or software that enables the integration code generator to simulate human interaction to interact with a desired interface.
  • the first simulated human interaction 626A may be implemented by simulating (e.g., via execution of event handlers, such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.) a pointer or mouse cursor.
  • the integration code generator may save the current state (or initial state) of the first interface at a first time 604 A and the initial second interface 612A.
  • the first interface at a first time 604A and the initial second interface 612A may be Internet markup language interfaces, in which an HTML representation of the first interface at a first time 604 A and the initial second interface 612A may be stored.
  • a representation of both the first interface at a first time 604A and the initial second interface 612A may be stored by the integration code generator.
  • the integration code generator may analyze the initial second interface 612A and determine that the element reading “Text 1, Option 1” corresponds to the element reading “Text 1,” the option selection element at the first time 622A reading “Option 1,” and the selectable element reading “Button 1” of the first interface at a first time 604 A.
  • the integration code generator may predict, as a result, that the option selection element at the first time 622A is a configuration option that, upon selection of the element “Button 1,” affects elements of the initial second interface 612A.
  • the integration code generator may utilize the second simulated human interaction 626B to select a different option from the option selection element, which may be denoted as “Option 2,” from the option selection element at the second time 622B.
  • the integration code generator may then utilize the second simulated human interaction 626B to select the element “Button 1 ”
  • the integration code generator may refresh the initial second interface 612A to obtain the refreshed second interface 612B.
  • the refreshed second interface 312B may be similar to one of the set of second interfaces 112 of FIG. 1 or one of the set of candidate interfaces 212 of FIG. 2.
  • the refreshed second interface 612B may be obtained by refreshing or otherwise updating the initial second interface 612A.
  • the integration code generator may compare the subsequent state of the refreshed second interface 612B with the stored representation (initial state) of the initial second interface 612A.
  • the integration code generator may determine that the interface elements at the second time 620B comprise an additional element reading “Text 1, Option 2” not present in the interface elements at the first time 620A, and further determine that the addition of the element “Text 1, Option 2” is a result of a configuration of the option selection element at the second time 622B selecting the element “Option 2” and the selection of the element reading “Button 1.”
  • the integration code generator may determine that the option selection element is a configuration element where, upon selection of a specific configuration and selection of the element reading “Button 1,” the specific configuration is displayed in the second interface.
  • first interfaces 604A-04B are shown for ease of explanation to only include a single option selection element, it is contemplated that a first interface may have multiple option selection elements of one or more types (e.g., drop-down lists, radio buttons, checkboxes, etc.).
  • the additional option selection elements may be identified, as with the first option selection element, based on classification data obtained by providing information about the first interface (e.g., source code of the first interface) as input to a set of machine learning classifiers.
  • Each of the additional option selection elements may be associated with their own set of additional options.
  • options for all of the option selection elements must be selected for the new listed selection to appear in the corresponding second interface, whereas in other cases only a subset of the option selection elements must have an option selected for the new listed selection to appear in the corresponding second interface.
  • the simulated human interaction may be performed to select an option of the additional option selection elements, and, further, the new listed selection element may include an indication that the options corresponding to the additional option selection elements were selected.
  • each different combination of options of the option selection elements may result in a new listed selection element upon engaging the button object and refresh of the second interface.
  • the integration code generator may generate integration code that models the elements of the various interfaces.
  • the integration code may be executable code that performs a customization process.
  • the integration code may comprise functions that allow an entity to streamline customization processes within one or more interface providers.
  • the interface provider may be a cinema reservation service.
  • the first interface may be a seat customization interface, comprising options selectable through the option selection element such as seat preference, heat preference, food preference, and/or variations thereof, which may be added with the element reading “Button 1 ”
  • the second interface may be an interface that, through the interface elements, displays chosen options.
  • the integration code generator may analyze and perform various processes with the interfaces of the cinema reservation service to determine integration code to reserve a movie with food and drink.
  • the integration code may cause an entity to select desired options through the option selection element, add the desired options through the “Button 1” element, and view the resulting interface elements of the second interface displaying the selected and added options.
  • Algorithm takes as input a first interface, a first button, and a second button, and identifiers for a particular element value, label, and digital image on the first interface, and a method for detecting the item elements on the second interface (as described in the present disclosure).
  • step 5 Interact with an “interactable” element in the first interface (as determined by a classifier (e.g., a heuristic)) and engage the first button again. a. If no more interactable elements remain, move to step 5.
  • a classifier e.g., a heuristic
  • FIG. 7 is a flowchart illustrating an example of a process 700 for identifying an element in one interface that is dependent upon a configuration of another element in another interface, in accordance with various embodiments.
  • Some or all of the process 700 may be performed under the control of one or more computer systems configured with executable instructions and/or other data, and may be implemented as executable instructions executing collectively on one or more processors.
  • the executable instructions and/or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer software program persistently stored on magnetic, optical, or flash media).
  • process 700 may be performed by any suitable system, such as the computing device 1300 of FIG. 13.
  • the process 700 includes a series of operations wherein a first user interface is executed and a first interface object that executes a second user interface is identified, a set of candidate second interface objects having multiple options are identified, and, for each of the second candidate interface objects, various options are selected, and a determination is made whether a new listed selection appears in a refreshed second user interface.
  • the candidate interface objects may be button objects (such as “submit” buttons) or other interface controls.
  • the system performing the process 700 may be a system such as the integration code generator 102 as described in connection with FIG. 1.
  • the system may execute a first user interface and identify a first interface object that executes a second user interface.
  • the first user interface may be an interface of an interface provider, such as the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2, which may provide various customizable services.
  • the system may identify the first interface object.
  • the system may analyze the first user interface to determine the first interface object.
  • the first interface object may cause the execution of a second user interface.
  • the process for identifying the first interface object may be similar to the processes 400 and 500 described in conjunction with FIGS. 4 and 5.
  • the first interface object may be identified through the usage of various classification algorithms/process to determine if the first interface object matches any known elements capable of executing the second user interface.
  • the system identifies a set of candidate second interface obj ects with multiple options.
  • the set of candidate second interface objects may be objects that allow selection of various options.
  • objects of the set of candidate second interface objects may be elements similar to the option selection element 622 as described in connection with FIG. 6.
  • the system may, in 706, iterate through the candidate second interface objects of the set of candidate second interface objects with multiple options.
  • the system may select a first option of a candidate second interface object.
  • the first option may be part of a set of options of the candidate second interface object.
  • the candidate second interface object may be an element denoting seat preferences for a cinema reservation, in which the set of options may be different seat locations.
  • the system may then, in 710, engage the first interface object to execute a second user interface in an initial state.
  • the system may save a representation of the initial state.
  • the system may select a second option of the candidate second interface object.
  • the system may refresh the second user interface.
  • the system may refresh the second user interface by engaging with the first interface object.
  • the system may refresh the second user interface by causing the second user interface to update through one or more processes, such as interacting with one or more interface providers.
  • the system may additionally save a representation of the updated, refreshed (or subsequent) state of the second user interface.
  • the system may determine differences between the refreshed state and the initial state of the second user interface.
  • the system may compare the refreshed state with the initial state. In some examples, the system may compare similar elements of the refreshed state with similar elements of the initial state, and analyze any potential differences.
  • the system performing the process 700 may return to 706 to process the next candidate second interface object of the set of candidate second interface objects with multiple options, and repeat the operations of 706-18 until the differences include the new listed selection and the appropriate candidate second interface object with multiple options is determined. On the other hand, if the differences include the new listed selection (e.g., the “Text 1, Option 2” of FIG.
  • the system may proceed to 720 to generate the integration code.
  • the new listed selection may include the selected second option.
  • the system may determine that the candidate second interface object with multiple options corresponds to a configuration of an element of the second user interface.
  • the system may generate integration code based on metadata about the new listed selection (e.g., object type, identifier, attributes, location in the second user interface source code, proximity to other elements, etc.) and the corresponding interface object that caused the new listed selection to appear in the second user interface.
  • the integration code may be executable code that performs a customization process (e.g., a process to automatically select options from option elements in accordance with a specified configuration).
  • the integration code may comprise functions that allow an entity to streamline customization processes within one or more interface providers.
  • the interface provider may be a cinema reservation service.
  • the first user interface may be a seat customization interface, comprising options selectable through the appropriate candidate second interface object such as seat preference, heat preference, food preference, and/or variations thereof, which may be confirmed with the first interface object.
  • the second user interface may be an interface that displays a confirmation of the chosen options.
  • the system may analyze and perform various processes with the interfaces of the cinema reservation service to determine integration code to reserve a movie with food and drink.
  • the integration code may cause an entity to select desired options through the appropriate candidate second interface object, add/confirm the desired options through the first interface object, and present the resulting second user interface to the entity.
  • the integration code may be implemented as a model or code that otherwise models the customization process.
  • the system may provide the integration code to a client device application.
  • the integration code is provided in response to a request by the client device (e.g., via an application executing on the client device) for the integration code for the interface provider.
  • the system may provide the integration code to one or more remote servers, which may provide the code to various devices and/or applications. Note that one or more of the operations performed in 702-22 may be performed in various orders and combinations, including in parallel.
  • FIG. 8 illustrates an example 800 of identifying an element in an interface that may have been added to a different interface, in accordance with an embodiment.
  • the example 800 includes a first interface 804, which may be interacted with through a simulated human interaction 826.
  • the example 800 also includes a second interface 812, which may be associated with the first interface 804, that comprises interface elements 820.
  • the first interface 804 may be an interface of an interface provider, which may provide various customizable services.
  • the first interface 804 may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the second interface 812 may be a different interface of the interface provider and may denote various information originating from the first interface 804.
  • the second interface 812 may be similar to one of the set of second interfaces 112 of FIG. 1 or one of the set of candidate interfaces 212 of FIG. 2.
  • the interface provider may be a library organization, in which the first interface 804 may be a listing of books, and the second interface 812 may denote books selected to be borrowed.
  • elements of the first interface 804 may be associated with elements of the second interface 812, and vice versa.
  • the selection of an element on the first interface 804 may affect a value displayed on the second interface 812 and/or interface elements 820.
  • an integration code generator such as the integration code generator 102 as described in connection with FIG. 1, to identify the process in which an element is added into another interface so as to generate integration code capable of interpreting and engaging with the various interface elements of the first interface and the second interface, the integration code generator 102 or similar entity may attempt to identify an element of the second interface that is dependent on an element of the first interface.
  • the integration code generator may perform one or more processes to obtain the first interface 804 and/or the second interface 812 (e.g., such as the processes 400 and 500 of FIGS. 4 and 5).
  • the integration code generator may be provided with the first interface 804 and the second interface 812.
  • the integration code generator may utilize an input from the entity to obtain the first interface 804.
  • the integration code generator may analyze the first interface 804.
  • the integration code generator may utilize various keyword searches, regular expressions, digital image recognition algorithms, neural networks, and/or machine learning algorithms to detect and identify elements of the first interface 804 (e.g., by passing a set of interface element nodes identified as input to one or more machine learning classifiers in order to classify interface element nodes).
  • the integration code generator may determine if any elements of the first interface 804 match any potentially known elements. The integration code generator may determine that the selection of one or more elements of the first interface 804 may result in the executing of the second interface 812, and utilize the one or more elements to obtain the second interface 812.
  • the integration code generator may then analyze the first interface 804 to determine a set of text/digital image pairs.
  • the integration code generator may utilize the source code of the first interface 804 to determine the set of text/digital image pairs.
  • the integration code generator may determine the set of text/digital image pairs based on any text/digital image pair elements of the first interface 804.
  • the integration code generator may further analyze the first interface 804 to determine one or more elements that may affect the second interface 812.
  • the integration code generator may utilize various classification processes to determine that the element reading “Button 1” of the first interface 804 may affect the second interface 812.
  • the element “Button 1” may be a button object or other interface control.
  • the integration code generator may utilize the simulated human interaction 826 to select the element reading “Button 1.”
  • the simulated human interaction 826 may be any hardware or software that enables the integration code generator to simulate human interaction to interact with a desired interface.
  • the simulated human interaction 826 may be implemented by simulating (e.g., via execution of event handlers, such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.) a pointer or mouse cursor.
  • the integration code generator may then refresh the second interface 812.
  • the integration code generator may refresh the second interface 812 through one or more processes that may refresh or otherwise update the second interface 812 to reflect any changes that may have occurred due to the simulated human interaction 826.
  • the integration code generator may analyze the interface elements 820 of the refreshed second interface 812 to determine all of the text/digital image pair elements present in the refreshed second interface 812. In various embodiments, the integration code generator may determine that the text/digital image pair element reading “Text 1” is present in the refreshed second interface 812. In some examples, the system may compare the refreshed second interface 812 to a previous, un-refreshed version of the second interface 812, which the system may have stored prior to refreshing the second interface 812, to determine if any new text/digital image pair elements have been generated. The integration code generator may iterate through the previously determined set of text/digital image pairs of the first interface 804, and determine if any text/digital image pairs of the first interface 804 match any text/digital image pair elements of the refreshed second interface 812.
  • the integration code generator may determine that the text/digital image pair element “Text 1” of the first interface 804 matches the text/digital image pair element “Text 1” of the second interface 812. The integration code generator may then further determine that the selection of the element reading “Button 1” of the first interface 804 caused the text/digital image pair element “Text 1” to be added to the second interface 812. The integration code generator may then generate integration code based upon the determined information.
  • integration code may be code that models the elements of the various interfaces.
  • the integration code may be executable code that may performs a process.
  • the integration code may comprise functions that allow an entity to streamline various processes within one or more interface providers.
  • the interface provider may be a library organization.
  • the first interface 804 may be a listing of books, which may be selected to be borrowed with the element reading “Button 1.”
  • the second interface 812 may be an interface that displays books selected to be borrowed.
  • the integration code generator may analyze and perform various processes with the interfaces of the library organization to determine integration code to select a book to be reserved.
  • the integration code may cause an entity to select a book, represented by a text/digital image pair element, add the selected element representing the book through the “Button 1” element, and view the resulting interface elements 820 of the second interface 812 displaying the selected book.
  • Algorithm III Location and Extraction of Item Elements in a User
  • Algorithm takes as input several first interfaces (at least 3). Start from one of these interfaces.
  • the first interfaces On the first of the first interfaces, identify all elements that approximately match a particular element type (this collection is referred to as the “particular element candidates”). These are elements on an interface, characterized by a CSS selector. These elements are akin to location descriptors - their value changes from first interface to another first interface, but their location is the same on all first interfaces (the particular element value of particular element changes from interface to interface, but the position of the element - e.g., the box where the particular element value is presented - is the same). These descriptors can be XPath selectors, or CSS selectors.
  • each “listed selection container” corresponds to the box containing information corresponding to a listed selection together with the corresponding price on a first interface from the matching in step 7). Since the information on the different first interfaces is different, each listed selection is matched to one first interface.
  • FIG. 9 is a flowchart illustrating an example of a process 900 for identifying an element in an interface that may have been added to a different interface, in accordance with various embodiments.
  • Some or all of the process 900 may be performed under the control of one or more computer systems configured with executable instructions and/or other data, and may be implemented as executable instructions executing collectively on one or more processors.
  • the executable instructions and/or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer software program persistently stored on magnetic, optical, or flash media).
  • process 900 may be performed by any suitable system, such as the computing device 1300 of FIG. 13.
  • the process 900 includes a series of operations wherein a first user interface having first and second interface objects is executed, a set of text/digital image pairs is identified, the first interface object is engaged to execute a second user interface, the second interface object is also engaged, the second user interface is then refreshed, and, for each of the text/digital image pairs, it is determined whether the text (and/or digital image) appears in the second user interface.
  • integration code is generated and provided in response to a request from an application running on a client device.
  • the system performing the process 900 may be a system such as the integration code generator 102 as described in connection with FIG. 1.
  • the system may execute a first user interface having first and second interface objects.
  • the first user interface may be an interface of an interface provider, such as the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2, which may provide various services as described in the present disclosure.
  • the system may identify the first and second interface objects.
  • the system may analyze the first user interface to determine the functions and identities of the first and second interface objects.
  • the first interface object may cause the execution of a second user interface.
  • the process for identifying the first interface object may be similar to the processes 400 and 500 described in conjunction with FIGS. 4 and 5.
  • the system may determine that the second interface object affects one or more elements of the second user interface.
  • the first and/or second interface objects may be identified through the usage of various classification algorithms/processes to determine if the objects match any known elements capable of executing the second user interface.
  • the system may identify a set of text/digital image pairs.
  • the system may analyze the first user interface and determine the set of text/digital image pairs.
  • a text/digital image pair is an element that comprises text paired with a digital image.
  • a digital image of a book accompanied with text denoting the name of the book may be considered a text/digital image pair.
  • a text/digital image pair may be a single element, combination of elements, and/or variations thereof.
  • the system may parse the source code of the first user interface to determine the set of text/digital image pairs.
  • the set is referred to as a set of pairs for ease of explanation, and it is contemplated that the set may comprise more than two dimensions.
  • the set may be of a set of text/digital image/value triplets, such as a digital image of a book, a name of the book, and a number of chapters of the book.
  • the types of data in the set may include any combination of text, digital images, or values, or even other media types as applicable (e.g., audio, video, etc.).
  • the system may engage the first interface object on the first user interface to execute the second user interface.
  • the system may engage with the first interface object through various processes, such as a simulated human interaction.
  • the system may then, in 908, engage the second interface object in the first user interface.
  • the system may have determined that the second interface object affects the second user interface.
  • the system may refresh the second user interface.
  • the system may refresh the second user interface by reloading the second user interface, obtaining an updated second user interface from one or more interface providers, or otherwise any process that may update the second user interface in response to engaging the second interface object.
  • the system may iterate through the previously determined set of text/digital image pairs.
  • the system may first analyze the second user interface to determine any text/digital image pair elements (or whatever combined elements of the set may be according to the particular implementation).
  • the system may analyze the second user interface and determine that after the second user interface was refreshed, a new text/digital image pair element has been generated and is present in the second user interface.
  • the system may compare the new text/digital image pair with the set of text/digital image pairs from the first user interface.
  • the system may return to 912 to repeat the operations of 912-14 until a match is found. On the other hand, if the new text/digital image pair does match a text/digital image pair from the set of text/digital image pairs from the first user interface, the system may proceed to 916.
  • the system may identify the matching text/digital image pair from the first user interface and the second user interface.
  • the system may further determine that engaging the second interface object in the first user interface caused the matching text/digital image pair from the first user interface to be generated in the second user interface.
  • the system may generate one or more models that may model this process and/or various other functionalities of the interfaces.
  • the system may then, in 916, generate integration code based upon the determined information (e.g., based on metadata about matching text/digital image pair (e.g., object types, identifiers, attributes, locations in the second user interface source code, proximity to other elements, etc.) in the second user interface).
  • the integration code as described throughout the present disclosure may be executable code that performs a selection process.
  • the integration code may comprise functions that allow an entity to streamline selection processes within one or more interface providers.
  • the interface provider may be a library organization.
  • the first user interface may be an interface depicting a listing of books, which may be selected to be borrowed with the second interface object.
  • the second user interface may be an interface that displays books selected to be borrowed.
  • the system may analyze and perform various processes with the interfaces of the library organization to determine integration code to select a book to be reserved.
  • the integration code may cause an entity to execute the first user interface, execute the second user interface through engaging the first interface obj ect, select a book represented by a digital image/text pair element, add the selected book through engaging the second interface object, and refresh and view the second user interface displaying the selected book.
  • the integration code may be implemented as a model or code that otherwise models the selection process.
  • the system may provide the integration code to a client device application.
  • the integration code is provided in response to a request by the client device (e.g., via an application executing on the client device) for the integration code for the interface provider.
  • the client device application may cause the generation of the integration code.
  • the system may provide the integration code to one or more remote servers, which may provide the code to various devices and/or applications.
  • the client device application may execute the integration code as part of one or more processes. Note that one or more of the operations performed in 902-18 may be performed in various orders and combinations, including in parallel.
  • the integration code may, upon execution, cause a device to engage with corresponding interface objects and user interfaces of different first interfaces by the same interface provider. Note, however, that the process 900 may need to be performed for multiple first user interfaces of the interface provider to obtain additional correlation that the correct elements corresponding to the matching text/digital image pairs have been identified in order to provide sufficient certainty that the correct interface objects have been identified.
  • FIG. 10 illustrates an example 1000 of identifying values in one interface that are dependent upon an element in another interface, in accordance with an embodiment.
  • the example 1000 includes a first interface at a first time 1004 A, which may be associated with an initial second interface 1012A.
  • the initial second interface 1012A may also include interface elements at the first time 1020A, which may be affected by a first simulated human interaction 1026A with the first interface.
  • the example 1000 also includes a first interface at a second time 1004B, which may be associated with a refreshed second interface 1012B.
  • the first interface at the second time 1004B may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the refreshed second interface 1012B may also include interface elements at the second time 1020B, which may be affected by a second simulated human interaction 1026B with the first interface.
  • the first interface at a first time 1004 A may be an interface of an interface provider, which may provide various customizable services as described in the present disclosure.
  • the first interface at the first time 1004 A may be similar to the first interface 104 of FIG. 1 or the initial interface 206 of FIG. 2.
  • the initial second interface 1012A may be a different interface of the interface provider, and may denote various information originating from the first interface at a first time 1004 A.
  • the interface provider may be a library organization, in which the first interface at a first time 1004A may be a book listing, comprising information regarding potential books such as number of chapters, genre, and/or variations thereof.
  • the initial second interface 1012A may be an interface that, through the interface elements at the first time 1020A, displays characteristics and quantities of selected books.
  • elements of the first interface at a first time 1004 A may be associated with elements of the initial second interface 1012A, and vice versa.
  • the selection of an element on the first interface at a first time 1004 A may affect a value displayed on the initial second interface and/or the interface elements at the first time 320A.
  • an integration code generator such as the integration code generator 102 as described in connection with FIG. 1, to generate integration code capable of interpreting and engaging with the various interface elements of the first interface and the second interface, the integration code generator 102 or similar entity may attempt to identify an element of the initial second interface 1012A that is dependent on an element of the first interface at a first time 1004 A.
  • the integration code generator may perform one or more processes to obtain the first interface at a first time 1004 A.
  • the integration code generator may be provided with the first interface at a first time 1004A and/or the initial second interface 1012A.
  • the integration code generator may utilize an input from the integration code generator to obtain the first interface at a first time 1004 A.
  • the integration code generator may analyze the first interface at a first time 1004 A.
  • the integration code generator may utilize various keyword searches, regular expressions, digital image recognition algorithms, neural networks, and/or machine learning algorithms to detect and identify elements of the first interface at a first time 1004A (e.g., by passing a set of interface element nodes identified as input to one or more machine learning classifiers in order to classify interface element nodes).
  • the integration code generator may determine if any elements of the first interface at a first time 1004 A match any potentially known elements.
  • the integration code generator may determine that the selection of one or more elements of the first interface at a first time 1004 A results in the initial second interface 1012A.
  • the integration code generator may additionally determine that the selectable element “Button 1” may affect the initial second interface 1012A.
  • the integration code generator may engage the element “Button 1” utilizing the first simulated human interaction 1026 A to produce the initial second interface 1012A.
  • the element “Button 1” may be a button object or other interface control.
  • the first simulated human interaction 1026 A may be hardware or software that enables the integration code generator to simulate human interaction to interact with a desired interface. In some examples, the first simulated human interaction 1026 A may be implemented by simulating (e.g., via execution of event handlers, such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.) a pointer or mouse cursor.
  • the integration code generator may save the current state of the initial second interface 1012A.
  • the first interface at a first time 1004 A and the initial second interface 1012A may be Internet markup language interfaces written in HTML.
  • a representation of both the initial second interface 1012A may be stored by the integration code generator.
  • the integration code generator may analyze the initial second interface 1012A and determine that the element depicting “Value C” corresponds to the element reading “Value A,” and that the first simulated human interaction 1026 A of the selectable element reading “Button 1” of the first interface at a first time 1004A caused the “Value C” to correspond with the value “A” (on refresh of the initial second interface 1012A).
  • the integration code generator may utilize the second simulated human interaction 1026B to engage the element “Button 1” again.
  • the integration code generator may refresh the initial second interface 1012A to obtain the refreshed second interface 1012B.
  • the refreshed second interface 1012B may be obtained by refreshing or otherwise updating the initial second interface 1012A.
  • the integration code generator may compare the refreshed second interface 1012B with the stored representation of the initial second interface 1012A.
  • the integration code generator may determine that the interface elements at the second time 1020B have been updated from the interface elements at the first time 1020 A, and further determine that the changes have resulted in the value “C” now being twice the value of “A” and that the value “D” has incremented by 1 as a result of the second engaging of the element reading “Button 1.” Consequently, the integration code generator may determine that the selection of the element reading “Button 1” adds the value specified in the element “Value A” to the value specified by the element “Value C,” and the value specified by the element “Value D” represents the quantity of element “Value A” that has been added by utilizing the element “Button 1.”
  • the integration code generator may generate integration code that models the elements of the various interfaces.
  • the integration code may be executable code that performs a process.
  • the integration code may comprise functions that allow an entity to streamline one or more processes within one or more interface providers.
  • the interface provider may be a library organization.
  • the first interface may be a book listing, comprising information regarding potential books such as number of chapters, genre, and/or variations thereof.
  • the second interface may be an interface that displays characteristics and quantities of selected books.
  • the value specified by element “Value D” may be the quantity of selected books
  • the value specified by element “Value C” may be the total number of chapters of the selected books.
  • the integration code generator may analyze and perform various processes with the interfaces of the library organization to determine integration code to select two copies of a book to be borrowed.
  • Algorithm takes as input a threshold, a first interface, a second button and a first button on this interface and a method for detecting listed selection elements on the second interface.
  • step 2) until threshold is reached (a threshold of 3 to 5 is enough in some cases).
  • FIG. 11 is a flowchart illustrating an example of a process 1100 for identifying values in one interface that are dependent upon an element in another interface, in accordance with various embodiments.
  • Some or all of the process 1100 may be performed under the control of one or more computer systems configured with executable instructions and/or other data, and may be implemented as executable instructions executing collectively on one or more processors.
  • the executable instructions and/or other data may be stored on a non-transitory computer-readable storage medium (e.g., a computer software program persistently stored on magnetic, optical, or flash media).
  • a non-transitory computer-readable storage medium e.g., a computer software program persistently stored on magnetic, optical, or flash media.
  • some or all of process 1100 may be performed by any suitable system, such as the computing device 1300 of FIG. 13.
  • the process 1100 includes a series of operations wherein a first user interface having first and second interfaces objects is executed, the first interface object is engaged, and then the second interface object is engaged to execute a second user interface, an initial state of the second user interface is obtained, the first interface object is engaged a second time, and then the second user interface is refreshed and compared with the initial state. Based on the comparison, locations of an element whose a value is associated with a value in the first user interface and an element whose value increments by 1 are determined, and integration code is generated based on the determination.
  • the system performing the process 1100 may be a system such as the integration code generator 102 as described in connection with FIG. 1.
  • the system may execute a first user interface, which may be denoted as a first user interface, having first and second interface objects.
  • the first user interface may be an interface of a service provider, which may provide various services as described in the present disclosure.
  • the system may identify the first and second interface objects.
  • the system may analyze the first user interface to determine the functions and identities of the first and second interface objects.
  • the second interface object may, as a result of being engaged (e.g., by simulating human interaction) cause the execution of a second user interface.
  • the system may determine that the first interface object affects one or more elements of the second user interface.
  • the first and/or second interface objects may be identified through the usage of various classification algorithms/processes to determine if the first and/or second interface objects match any potentially known elements.
  • the system may engage the first interface object in the first user interface.
  • the system may simulate human interaction to engage with the first interference object.
  • simulated human interaction refers to hardware or software that causes the system to simulate human interaction with a particular interface.
  • simulated human interaction may be implemented by simulating (e.g., via execution of event handlers, such as onClick, onMouseOver, onMouseOut, onKeyDown, etc.) a pointer or mouse cursor.
  • the system may engage the second interface object on the first user interface to execute the second user interface in an initial state.
  • the system may engage the second interface object by simulating human interaction.
  • the system may engage the first interface object in the first user interface a second time.
  • the system may refresh the second user interface to a refreshed state.
  • the system may update the second user interface to obtain the refreshed state of the second user interface.
  • the system may then analyze the initial state and refreshed state of the second user interface.
  • the system may determine a set of first values of the second user interface.
  • the set of first values may be present in both the initial state and the refreshed state of the second user interface, although in some examples, one or more values of the set of first values may be different between the initial state and the refreshed state of the second user interface.
  • the system may identify differences between values of the set of first values in the second user interface in the initial state and the refreshed state. In some examples, the system may compare a value in the refreshed state with a value in the initial state to determine a difference of the values.
  • the system may compare a value of the set of first values in the initial state with each value of the set of first values in the refreshed state, and vice versa, to identify differences.
  • the system may, in 1114, iterate through the first/next differences of the values.
  • the system may determine whether the difference between the value of the set of first values is equal to the value 1. If the difference is not equal to the value of 1, the system performing the process 1100 may return to 1112 repeat the operations of 1112-16 until a difference of 1 is found. On the other hand, if the system identifies a difference with the value of 1, the system may proceed to 1118.
  • the system may obtain locations of the first values associated with the difference and a second value associated with the first values.
  • the system may determine that the second value may be associated with the first values such that the second value may change with the first value.
  • the system may determine that the first interface object in the first user interface may cause the second value to change along with the first values.
  • the system may determine that engaging the first interface object causes the first values to increase by 1 and the second value to increase by a fixed quantity.
  • the system may further determine that the first values represent a quantity of an element, and the second values represent a value of the element.
  • the system may generate one or more models that may model the interface objects and/or various other functionalities of the interfaces.
  • the system may generate integration code based on metadata about the element corresponding to the quantity and listed selection value (e.g., object types, identifiers, attributes, locations in the second user interface source code, proximity to other elements, etc.) in the second user interface.
  • the integration code may be based upon the determined information.
  • the integration code may be executable code that performs one or more processes.
  • the integration code may comprise functions that allow an entity to streamline processes within one or more interface providers.
  • the interface provider may be a library organization.
  • the first interface may be a book listing, comprising information regarding potential books such as number of chapters, genre, and/or variations thereof.
  • the second interface may be an interface that displays characteristics and quantities of selected books.
  • the first value may be the quantity of selected books
  • the second value may be the total number of chapters of the selected books.
  • the system may analyze and perform various processes with the interfaces of the library organization to determine integration code to select two copies of a book to be borrowed.
  • the integration code may cause an entity to determine the desired book, add the desired book twice by engaging with the first interface object, and view the second user interface displaying the added books.
  • the integration code may be implemented as a model or code that otherwise models the selection process.
  • the system may provide the integration code to a client device application.
  • the client device application may cause the generation of the integration code.
  • the system may provide the integration code to one or more remote servers, which may provide the code to various devices and/or applications.
  • the client device application may execute the integration code as part of one or more processes. Note that one or more of the operations performed in 1102-1122 may be performed in various orders and combinations, including in parallel.
  • the integration code may, upon execution, cause a device to engage with corresponding interface objects and user interfaces of different first interfaces by the same interface provider. Note, however, that the process 1100 may need to be performed for multiple first user interfaces of the interface provider to obtain additional correlation that the correct elements corresponding to the quantity and listed selection value have been identified in order to provide sufficient certainty that the correct interface objects have been identified.
  • FIG. 12 illustrates an example interface 1200 of an application running on a client device 1228.
  • the example interface 1200 may comprise interface elements 1222, including a bookmark interface element 1224.
  • the interface elements 1222 may be elements that allow an entity to interact/engage with the example interface 1200.
  • the bookmark interface element 1224 may perform various functionalities, such as saving a state of a configuration of the interface elements 1222, transmitting a state of a configuration of the interface elements 1222 to one or more systems for further processing, and/or variations thereof.
  • the client device 1228 may be a computing device, such as the client device 216 of FIG. 2 or the computing device 1300 of FIG.
  • the application may be a software application, such as a browser, capable of causing the execution of integration code described in the present disclosure.
  • the application may, as a result of running on the client device 1228, display the example interface 1200 of the interface provider on the client device 1228, and may further display one or more of its own graphical overlays on top of or in lieu of the example interface 1200 of the interface provider.
  • the example interface 1200 may be an interface of an interface provider, which may provide various services as described in the present disclosure and the interface elements 1222 may allow an entity to select and configure various elements.
  • the application running on the client device 1228 may obtain integration code from a system, such as the integration code generator 102 as described in connection FIG. 1, that, upon execution by the client device 1228 may streamline one or more processes of involving the example interface 1200. For example, execution of the integration code via the application running on the client device 1228 may cause the client device to interact with the interface element 1222 of the interface provider.
  • the entity may utilize the integration code to perform one or more processes.
  • the one or more processes may comprise executing the integration code to cause selection of an option from the “Dropdown 1” element, engaging the “Button 1” and “Button 2” elements, and selection of the bookmark interface element 1224.
  • the “Button 1” and “Button 2” may be button objects or other interface controls.
  • the bookmark interface element may be an interface element that, in response to being engaged, may cause the client device to bookmark, save, or otherwise store the selected elements/configurations.
  • the client device may request integration code for a particular interface provider and the system (e.g., the integration code generator) may not yet have integration code generated for that particular interface provider; in such a case, the system may perform one or more of the processes described in the present disclosure (e.g., processes 400, 500, 700, 900, and 1100 of FIGS. 4, 5, 7, 9, and 11 respectively) in order to generate and provide the integration code to the client device 1228 in real-time.
  • the system e.g., the integration code generator
  • the system may perform one or more of the processes described in the present disclosure (e.g., processes 400, 500, 700, 900, and 1100 of FIGS. 4, 5, 7, 9, and 11 respectively) in order to generate and provide the integration code to the client device 1228 in real-time.
  • the interface provider may be a library organization.
  • the example interface 1200 may be a markup language interface of the library organization that allows an entity to select and borrow a book.
  • a user may desire to select and borrow a specific book and may make the appropriate selections of the interface elements 1222 to add the book to a queue. In this example, however, the user may decide to wait and finish the borrowing process for the book at a different time.
  • the user may engage the bookmark interface element 1224 to store the current configuration information in volatile storage such as random access memory (RAM), or non-volatile (i.e., persistent) storage, such as flash memory or magnetic media.
  • volatile storage such as random access memory (RAM), or non-volatile (i.e., persistent) storage, such as flash memory or magnetic media.
  • such configuration information may be stored in persistent storage on the client device or at a server of an entity that hosts the integration code generator.
  • the user may decide to resume the borrowing process and restore the selection from storage at the different time; however, at the different time, the previous session with the library organization may be expired. Consequently, the application (via the client device) may request integration code from a system such as the integration code generator 102 of FIG. 1, whereupon the system may generate (if not already generated) integration code for the library organization and provide the integration code to the client device.
  • Execution of the integration code by the client device may cause the client device to retrieve the configuration information previously stored in response to the user engaging the bookmark interface element 1224 and dynamically re-select the appropriate interface element options in accordance with the configuration information.
  • the re-selection of the appropriate interface elements may be hidden by an overlay of the application, may be performed in a background process of the client device 1228, or otherwise be hidden from the user.
  • the application running on a first client device may allow the stored configuration information to be shared with an additional user even for interfaces that lack such intrinsic sharing capability.
  • the other user may download and install an additional copy of the application to a second client device, and the application may share the stored information with the additional application running on the second client device.
  • stored configuration information may be shared by sending a link (e.g., a Uniform Resource Identifier) in a message (e.g., text message, e-mail message, etc.) to an address associated with the second client device.
  • a link e.g., a Uniform Resource Identifier
  • Engaging with e.g., clicking, touching, or otherwise selecting the link
  • the additional application may restore the selection and resume the process in a similar manner as the first client device could have.
  • the example interface 1200 may be a simplified interface to an interface provider, and may be generated by the application running on the client device 1228.
  • the example interface 1200 may comprise various elements of one or more interfaces of the interface provider, and may provide a simplified view of the one or more interfaces.
  • the system may have previously generated sets of integration codes to execute various processes utilizing the one or more interfaces of the interface provider.
  • an entity may utilize the interface elements 1222 to select a configuration from the “Dropdown 1” element, engage with the “Button 1” and “Button 2” elements, and select the bookmark interface element 1224.
  • the system may retrieve the appropriate integration code, and execute one or more processes to complete various tasks which may be specified by engaging the interface elements 1222 by the entity.
  • the interface provider may be a library organization.
  • the example interface 1200 may provide a simplified view of one or more interfaces of the library organization.
  • the example interface 1200 may provide the essential elements to select and borrow a book through the interface elements 1222.
  • an entity may utilize the interface elements 1222 to select a book to be borrowed.
  • the system may utilize the inputs to the interface elements 1222 to perform one or more processes, which may be executed through generated and/or retrieved integration codes, to select and borrow the book utilizing one or more other interfaces of the library organization on behalf of the entity.
  • the entity may decide to borrow the book at a later time, and utilize the bookmark interface element 1224 to bookmark the specific book.
  • the system may store any integration codes that may potentially be utilized to borrow the specific book, and utilize the integration codes at the later time.
  • the device that utilizes the integration code need not be limited to a client devices, but may instead be a server of the entity that generated the integration code or even a third party entity.
  • the integration code may be retrieved and executed in a headless browser application running on a distributed server hosted by a service provider that generated the integration code.
  • sets of integration code for one or more interface providers may be executed in a batch process on a backend server of an entity that generated the integration code. For example, if an individual has bookmarked 20 books to reserve, and those 20 books are distributed among eight different libraries each associated with different integration code, the individual may select a “Reserve All” control object from an interface on the individual’s client device to cause the client device to send a request to a backend server to cause the backend server to execute the sets of integration code corresponding to the eight different libraries in order to reserve the bookmarked books at their respective library.
  • FIG. 13 is an illustrative, simplified block diagram of a computing device 1300 that can be used to practice at least one embodiment of the present disclosure.
  • the computing device 1300 includes any appropriate device operable to send and/or receive requests, messages, or information over an appropriate network and convey information back to a user of the device.
  • the computing device 1300 may be used to implement any of the systems illustrated and described above.
  • the computing device 1300 may be configured for use as a data server, a web server, a portable computing device, a personal computer, a cellular or other mobile phone, a handheld messaging device, a laptop computer, a tablet computer, a set-top box, a personal data assistant, an embedded computer system, an electronic book reader, or any electronic computing device.
  • the computing device 1300 may be implemented as a hardware device, a virtual computer system, or one or more programming modules executed on a computer system, and/or as another device configured with hardware and/or software to receive and respond to communications (e.g., web service application programming interface (API) requests) over a network.
  • communications e.g., web service application programming interface (API) requests
  • the computing device 1300 may include one or more processors 1302 that, in embodiments, communicate with and are operatively coupled to a number of peripheral subsystems via a bus subsystem.
  • these peripheral subsystems include a storage subsystem 1306, comprising a memory subsystem 1308 and a file/disk storage subsystem 1310, one or more user interface input devices 1312, one or more user interface output devices 1314, and a network interface subsystem 1316.
  • Such storage subsystem 1306 may be used for temporary or long-term storage of information.
  • the bus subsystem 1304 may provide a mechanism for enabling the various components and subsystems of computing device 1300 to communicate with each other as intended. Although the bus subsystem 1304 is shown schematically as a single bus, alternative embodiments of the bus subsystem utilize multiple buses.
  • the network interface subsystem 1316 may provide an interface to other computing devices and networks.
  • the network interface subsystem 1316 may serve as an interface for receiving data from and transmitting data to other systems from the computing device 1300.
  • the bus subsystem 1304 is utilized for communicating data such as details, search terms, and so on.
  • the network interface subsystem 1316 may communicate via any appropriate network that would be familiar to those skilled in the art for supporting communications using any of a variety of commercially available protocols, such as Transmission Control Protocol/Intemet Protocol (TCP/IP), User Datagram Protocol (UDP), protocols operating in various layers of the Open System Interconnection (OSI) model, File Transfer Protocol (FTP), Universal Plug and Play (UpnP), Network File System (NFS), Common Internet File System (CIFS), and other protocols.
  • TCP/IP Transmission Control Protocol/Intemet Protocol
  • UDP User Datagram Protocol
  • OSI Open System Interconnection
  • FTP File Transfer Protocol
  • UFP Universal Plug and Play
  • NFS Network File System
  • CIFS Common Internet File System
  • the network in an embodiment, is a local area network, a wide-area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, a cellular network, an infrared network, a wireless network, a satellite network, or any other such network and/or combination thereof, and components used for such a system may depend at least in part upon the type of network and/or system selected.
  • a connection-oriented protocol is used to communicate between network endpoints such that the connection-oriented protocol (sometimes called a connection-based protocol) is capable of transmitting data in an ordered stream.
  • a connection-oriented protocol can be reliable or unreliable.
  • the TCP protocol is a reliable connection-oriented protocol.
  • Asynchronous Transfer Mode (ATM) and Frame Relay are unreliable connection-oriented protocols. Connection-oriented protocols are in contrast to packet-oriented protocols such as UDP that transmit packets without a guaranteed ordering. Many protocols and components for communicating via such a network are well known and will not be discussed in detail. In an embodiment, communication via the network interface subsystem 1316 is enabled by wired and/or wireless connections and combinations thereof.
  • the user interface input devices 1312 includes one or more user input devices such as a keyboard; pointing devices such as an integrated mouse, trackball, touchpad, or graphics tablet; a scanner; a barcode scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems, microphones; image sensors or other digital image capture devices, and other types of input devices.
  • user input devices such as a keyboard; pointing devices such as an integrated mouse, trackball, touchpad, or graphics tablet; a scanner; a barcode scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems, microphones; image sensors or other digital image capture devices, and other types of input devices.
  • the one or more user interface output devices 1314 include a display subsystem, a printer, or non-visual displays such as audio output devices, etc.
  • the display subsystem includes a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), light emitting diode (LED) display, or a projection or other display device.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • LED light emitting diode
  • output device is intended to include all possible types of devices and mechanisms for outputting information from the computing device 1300.
  • the one or more user interface output devices 1314 can be used, for example, to present user interfaces to facilitate user interaction with applications performing processes described and variations therein, when such interaction may be appropriate.
  • the storage subsystem 1306 provides a computer-readable storage medium for storing thereon the basic programming and data constructs that provide the functionality of at least one embodiment of the present disclosure.
  • the applications software programs, code modules, executable instructions
  • the storage subsystem 1306 additionally provides a repository for storing thereon data used in accordance with the present disclosure.
  • the storage subsystem 1306 comprises a memory subsystem 1308 and a file/disk storage subsystem 1310.
  • the memory subsystem 1308 includes a number of memories, such as a main random access memory (RAM) 1318 for storage of executable instructions and data during software program execution and/or a read only memory (ROM) 1320, in which fixed executable instructions can be stored.
  • the file/disk storage subsystem 1310 provides a non-transitory persistent (non-volatile) storage for software program and data files and can include a hard disk drive, a floppy disk drive along with associated removable media, a Compact Disk Read Only Memory (CD-ROM) drive, an optical drive, removable media cartridges, or other like storage media.
  • CD-ROM Compact Disk Read Only Memory
  • the computing device 1300 includes at least one local clock 1324.
  • the at least one local clock 1324 in some embodiments, is a counter that represents the number of ticks that have transpired from a particular starting date and, in some embodiments, is located integrally within the computing device 1300.
  • the at least one local clock 1324 is used to synchronize data transfers in the processors for the computing device 1300 and the subsystems included therein at specific clock pulses and can be used to coordinate synchronous operations between the computing device 1300 and other systems in a data center.
  • the local clock is a programmable interval timer.
  • the computing device 1300 could be of any of a variety of types, including a portable computer device, tablet computer, a workstation, or any other device described below. Additionally, the computing device 1300 can include another device that, in some embodiments, can be connected to the computing device 1300 through one or more ports (e.g., USB, a headphone jack, Lightning connector, etc.). In embodiments, such a device includes a port that accepts a fiber-optic connector. Accordingly, in some embodiments, this device is that converts optical signals to electrical signals that are transmitted through the port connecting the device to the computing device 1300 for processing. Due to the ever-changing nature of computers and networks, the description of the computing device 1300 depicted in FIG.
  • data may be stored in a data store (not depicted).
  • a “data store” refers to any device or combination of devices capable of storing, accessing, and retrieving data, which may include any combination and number of data servers, databases, data storage devices, and data storage media, in any standard, distributed, virtual, or clustered system.
  • a data store in an embodiment, communicates with block-level and/or object level interfaces.
  • the computing device 1300 may include any appropriate hardware, software and firmware for integrating with a data store as needed to execute aspects of one or more applications for the computing device 1300 to handle some or all of the data access and business logic for the one or more applications.
  • the data store includes several separate data tables, databases, data records, data files, dynamic data storage schemes, and/or other data storage mechanisms and media for storing thereon data relating to a particular aspect of the present disclosure.
  • the computing device 1300 includes a variety of data stores and other memory and storage media as discussed above. These can reside in a variety of locations, such as on a storage medium local to (and/or resident in) one or more of the computers or remote from any or all of the computers across a network.
  • the information resides in a storage-area network (SAN) familiar to those skilled in the art, and, similarly, any necessary files for performing the functions attributed to the computers, servers or other network devices are stored locally and/or remotely, as appropriate.
  • SAN storage-area network
  • the computing device 1300 may provide access to data including, but not limited to, text, graphics, audio, video, and/or other data that is provided to a user in the form of Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript, Cascading Style Sheets (CSS), JavaScript Object Notation (JSON), and/or another appropriate language.
  • HTML Hypertext Markup Language
  • XML Extensible Markup Language
  • JavaScript JavaScript
  • CSS Cascading Style Sheets
  • JSON JavaScript Object Notation
  • the computing device 1300 may provide the data in one or more forms including, but not limited to, forms that are perceptible to the user audibly, visually, and/or through other senses.
  • PHP Hypertext Preprocessor
  • Python Python
  • Ruby Ruby
  • Perl Java
  • HTML Hypertext Preprocessor
  • XML HyperText Markup Language
  • JSON Java
  • operations described as being performed by a single device are performed collectively by multiple devices that form a distributed and/or virtual system.
  • the computing device 1300 typically will include an operating system that provides executable software program instructions for the general administration and operation of the computing device 1300 and includes a computer-readable storage medium (e.g., a hard disk, random access memory (RAM), read only memory (ROM), etc.) storing thereon executable instructions that if executed (e.g., as a result of being executed) by a processor of the computing device 1300 cause or otherwise allow the computing device 1300 to perform its intended functions (e.g., the functions are performed as a result of one or more processors of the computing device 1300 executing instructions stored on a computer-readable storage medium).
  • a computer-readable storage medium e.g., a hard disk, random access memory (RAM), read only memory (ROM), etc.
  • the computing device 1300 operates as a web server that runs one or more of a variety of server or mid-tier applications, including Hypertext Transfer Protocol (HTTP) servers, FTP servers, Common Gateway Interface (CGI) servers, data servers, Java servers, Apache servers, and business application servers.
  • HTTP Hypertext Transfer Protocol
  • CGI Common Gateway Interface
  • computing device 1300 is also capable of executing software programs or scripts in response to requests from user devices, such as by executing one or more web applications that are implemented as one or more scripts or software programs written in any programming language, such as Java®, C, C# or C++, or any scripting language, such as Ruby, PHP, Perl, Python, or TCL, as well as combinations thereof.
  • the computing device 1300 is capable of storing, retrieving, and accessing structured or unstructured data.
  • computing device 1300 additionally or alternatively implements a database, such as one of those commercially available from Oracle®, Microsoft®, Sybase®, and IBM® as well as open-source servers such as MySQL, Postgres, SQLite, MongoDB.
  • the database includes table-based servers, document-based servers, unstructured servers, relational servers, non-relational servers, or combinations of these and/or other database servers.
  • a computer-implemented method comprising: determining a set of first button object candidates in a first user interface; determining a set of second button object candidates in the first user interface, the set of second button object candidates associated with a respective set of values; executing a set of candidate user interfaces by simulating human interaction with the set of first button object candidates; producing a set of candidate rankings by at least, for each second button object of the set of second button object candidates: storing a set of respective initial states of the set of candidate user interfaces; simulating the human interaction with the second button object; refreshing the set of candidate user interfaces; and for each candidate user interface of the set of candidate user interfaces, on a condition that a difference between a respective initial state of the candidate user interface and a current state of the candidate user interface has a correlation to a respective value associated with the second button object, increasing candidate rankings of the second button object and the candidate user interface; designating, based on the set of candidate rankings, a principal second button object of the set of second button object candidates and a principal user
  • a system comprising: one or more processors; and memory including executable instructions that, if executed by the one or more processors, cause the system to: access a first interface; obtain initial data of a second interface different from the first interface; engage a first object in the first interface that is associated with a first value; obtain refreshed data of the second interface; determine, at least in part as a result of a difference between the initial data and the refreshed data corresponding to the first value, a correlation between the first value and a second object in the refreshed data of the second interface; determine, based on the correlation, that the second object is a target object and the second interface is a target interface; and generate, based on a location of the target object in the second interface, integration code that, as a result of execution by a device, causes the device to: engage the first object; engage a second object in the first interface to execute the target interface; and extract, from the target interface, a target value from the target object in the target interface.
  • executable instructions that cause the system to obtain the initial data further include instructions that further cause the system to: engage the second object in the first interface to cause the system to execute the second interface; and obtain the initial data from source code of the second interface.
  • the executable instructions further include instructions that further cause the system to: provide an application to the device; and provide the integration code to the application; and the execution of the integration code by the device is performed as a result of execution of the application by the device.
  • the executable instructions further include instructions that further cause the system to: access a third interface; engage a third object in the third interface that is associated with a second value; obtain second refreshed data of the second interface; determine, at least in part on the second refreshed data and the second value, another correlation between the second value and the target object in the second interface is the target interface; and the executable instructions that cause the system to determine that the second object is the target object and that the second interface is the target interface further include instructions that further cause the system to determine, further based on the additional correlation, that the second object is the target object and that the second interface is the target interface.
  • executable instructions further include instructions that further cause the system to: identify a set of candidate second objects in the first user interface; and determine the second object from the set of candidate second objects.
  • executable instructions that cause the system to determine the second object further include instructions that further cause the system to determine, at least in part as a result of a determination that engaging a candidate second object of the set of candidate second objects causes execution of the second interface, that the candidate second object is the second object.
  • executable instructions that cause the system to identify the set of candidate second objects further include instructions that further cause the system to: identify a set of objects within software program code of the first interface; classify, using one or more machine learning classifiers, the set of objects to produce a set of classified objects with rankings in accordance with respective likelihoods of being the second object; and determine, based on the rankings, the set of candidate second objects from the set of classified objects.
  • a non-transitory computer-readable storage medium storing thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least: determine a set of first control candidates in a first user interface; engage the set of first control candidates to execute a set of candidate user interfaces; store a set of initial states of the set of candidate user interfaces; determine, in the first user interface, a first value associated with a second control; engage the second control; refresh the set of candidate user interfaces; compare a set of current states of the set of candidate user interfaces with the set of initial states to: determine, based on differences between the set of initial states and the set of current states, a target user interface of the set of candidate user interfaces that includes a second value that has been incremented by an amount that is correlated to the first value; and set a control candidate of the set of first control candidates that caused the final user interface to execute to be a final control; determine metadata based on: the second control; the final control; and the final user interface; and generate
  • executable instructions further include instructions that further cause the computer system to: identify a set of objects within the first user interface; classify, using one or more heuristics, the set of objects to produce a set of classified objects with rankings in accordance with respective likelihoods of being a value associated with the second control; and determine, based on the rankings, that the first value is the value associated with the second control.
  • a computer-implemented method comprising: executing a first user interface having a first interface object, a second interface object, and a third interface object, the third interface object associated with a first option element and a second option element; setting the third interface object to the first option element; simulating human interaction with the second interface object a first time; executing a second user interface as a result of simulating the human interaction with the first interface object, the second user interface being in a first state that includes a first listed selection corresponding to the first option element; setting the third interface object to the second option element; simulating the human interaction with the second interface object a second time; refreshing the second user interface to produce the second user interface in a second state; as a result of verifying that a difference between the second state and the first state includes a second listed selection corresponding to the second option element, generating integration code; and as a result of execution of the integration code by a client device, selecting one of the first option element or the second option element of the third interface object of the first user interface.
  • a system comprising: one or more processors; and memory including executable instructions that, if executed by the one or more processors, cause the system to: execute a first interface having a first control object, a second control object, and a third control object, the third control object associated with a first option and a second option; select the first option of the third control object; engage the second control object a first time; engage the first control object to execute a second interface in a first state, the second interface in the first state including a first data item associated with the first option; select the second option of the third control object; engage the second control object a second time; re-execute the second interface to produce the second interface in a second state; generate integration code as a result of a verification that a difference between the second state and the first state includes a second data item associated with the second option; and provide, to a device, the integration code that, as a result of execution of the integration code by the device, causes the device to select one of the first option or the second option of the third control object
  • executable instructions further include instructions that further cause the system to: identify a set of document object model (DOM) element nodes in the first interface; and determine the first control object, the second control object, and the third control object from the set of DOM element nodes.
  • DOM document object model
  • the executable instructions further include instructions that further cause the system to provide metadata about the set of DOM element nodes to the set of machine learning classifiers to produce a set of classifications of the set of DOM element nodes; and the executable instructions that cause the system to determine the first control object, the second control object, and the third control object further include instructions that further cause the system to determine the first control object, the second control object, and the third control object in the first interface based on the set of classifications.
  • the integration code further causes the device to further display the first option and the second option of the third control object for selection by a user of the device; and the integration code that causes the device to select one of the first option or the second option of the first interface is performed as a result of the device receiving a selection of one of the first option or the second option from a user via the software application.
  • the executable instructions further include instructions that further cause the system to provide an additional software application to an additional device; the software application further causes the device to: generate data that includes: a Uniform Resource Identifier (URI) associated with the first interface; and an indication of the selection; and provide the data to the additional device; and in response to receipt of the data, the additional software application, as a result of execution by the additional device, causes the additional device to: obtain the first interface associated with the URI; and select one of the first option or the second option of the third control object of the first interface.
  • URI Uniform Resource Identifier
  • a non-transitory computer-readable storage medium storing thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least: execute a first user interface having a first control, a second control, and a third control, the third control associated with a first option and a second option; select the first option of the third control; engage the second control a first time; engage the first control to execute a second user interface in a first state, the second user interface in the first state including a first listed selection associated with the first option; select the second option of the third control; engage the second control a second time; refresh the second user interface to produce the second user interface in a second state; generate integration code as a result of a verification that a difference between the second state and the first state includes a second listed selection associated with the second option different from the first listed selection; and as a result of execution of the integration code by a computing device, select one of the first option or the second option of the third control of the first user interface.
  • executable instructions further include instructions that further cause the computer system to: identify a first candidate and a second candidate for the third control; select a first candidate option of the first candidate; engage the second control at an initial time; identify an initial listed selection within the second user interface at an initial state; select a second candidate option of the first candidate at a subsequent time after the initial time; engage the second control after the subsequent time; refresh the second user interface to produce the second user interface at a subsequent state; and as a result of a failure to verify that the subsequent state includes a second listed selection associated with the second candidate option, determine that the second candidate is the third control.
  • the executable instructions further include instructions that further cause the computer system to provide a software application to the computing device; and the software application, as a result of being executed by the computing device, causes the computing device to receive the integration code to be processed by the software application.
  • software application further causes the computing device to: receive a selection of one of the first option or the second option from a user via the software application; and store the selection in persistent storage of the computing device.
  • executable instructions further include instructions that further cause the computer system to: extract a set of document object model (DOM) element nodes from source code of the first user interface; provide the set of DOM element nodes as input to at least one machine learning classifier to produce classification data indicating likelihood of each of the set of DOM element nodes being associated with option elements; and identifying the third control based at least in part on the classification data.
  • DOM document object model
  • a computer-implemented method comprising: obtaining user interface source code; locating a particular object in the user interface source code by at least: identifying a set of objects in the user interface source code; passing each of the set of objects as input to a machine learning classifier to determine a set of candidate objects for the particular object; for each candidate object of the set of candidate objects: simulating human interaction with the candidate object; detecting a result of simulating the human interaction; and modifying a ranking of the set of candidate objects based on the result to produce a modified set of candidate objects; and determining, based on the ranking, the particular object from the modified set of candidate objects; causing a user interface to be displayed on a client device; and as a result of a selection made via the user interface, simulating the human interaction with the particular object.
  • the computer-implemented method further includes, further for each candidate object of the set of candidate objects, examining additional user interface source code in an initial state; and detecting the result of simulating the human interaction includes: refreshing a user interface associated with the additional user interface source code; obtaining the additional user interface source code in a refreshed state; and determining that a difference between the additional user interface source code in the initial state and the additional user interface source code in the refreshed state includes a value associated with the candidate object.
  • identifying the set of objects includes identifying the set of objects from at least one of: an object model of the user interface source code,
  • JavaScript in the user interface source code or an image recognition algorithm.
  • a system comprising: one or more processors; and memory including executable instructions that, if executed by the one or more processors, cause the system to: obtain software program code of an interface; identify a particular object in the software program code by at least causing the system to: identify a set of candidate objects in the software program code; for each candidate object of the set of candidate objects, produce a modified set of candidate objects by causing the system to at least: simulate human interaction with the candidate object; detect a result of simulating the human interaction; and modify a ranking of the candidate object based on the result; and determine, based on the ranking, the particular object from the modified set of candidate objects; generate integration code based on the particular object; and provide, to a device, the integration code to cause, as a result of execution by the device, the device to engage the particular object in the interface.
  • executable instructions that cause the system to identify the set of candidate objects in the software program code further include instructions that further cause the system to identify the set of candidate objects in the software program code using at least one of: a keyword search, a regular expression, or digital image recognition.
  • the executable instructions further include instructions that further cause the system to, further for each candidate object of the set of candidate objects, examining additional interface software program code in an initial state; and the executable instructions that cause the system to detect the result of simulating the human interaction further include instructions that further cause the system to: re-execute an additional interface associated with the additional interface software program code; obtain the additional interface software program code in a refreshed state; and determine the result based on whether a difference between the additional interface software program code in the initial state and the additional interface software program code in the refreshed state includes a value associated with the candidate object.
  • executable instructions that cause the system to modify the ranking further include instructions that further cause the system to increase the ranking of the candidate object if the difference includes the value.
  • executable instructions further include instructions that further cause the system to provide, to the device, a software application that, as a result of being executed by the device, causes the device to execute the integration code.
  • a non-transitory computer-readable storage medium storing thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least: obtain executable source code of a user interface; identify a particular object in the executable source code by at least causing the computer system to: identify a set of objects in the executable source code; provide the set of objects as input to one or more machine learning classifiers to produce a set of candidate objects; for each candidate object of the set of candidate objects: simulate human interaction with the candidate object; detect a result of simulating the human interaction; and modify a ranking of the set of candidate objects based on the result to produce a modified set of candidate objects; and determine, based on the ranking, the particular object from the modified set of candidate objects; generate integration code based on the particular object; and provide the integration code to a computing device to cause, as a result of execution of the integration code by the computing device, the computing device to engage the particular object from the user interface.
  • executable instructions that cause the computer system to provide the set of objects as input to the one or more machine learning classifiers further include instructions that further cause the computer system to, for each object of the set of objects, provide metadata about the object to at least one of the one or more machine learning classifiers, the metadata including a set of inputs associated with the object.
  • the user interface is provided by a third party; and the executable instructions further include instructions that further cause the computer system to: obtain additional software program code of an additional interface provided by an additional third party; identify an additional particular object in the additional software program code; generate additional integration code based on the additional particular object; and as a result of execution of the additional integration code by the computing device, engage the additional particular object in the additional interface.
  • non-transitory computer-readable storage medium of any of clauses 54-59 wherein: the user interface is a first user interface; and the executable instructions that cause the computer system to detect the result further include instructions that further cause the computer system to: refresh a second user interface; and determine, as the result, whether a difference between a current state of the second user interface and a previous state of the second user interface corresponds to the candidate object of the first user interface.
  • the executable instructions that cause the computer system to detect the result further include instructions that further cause the computer system to determine whether an additional user interface executes as the result of simulating the human interaction.
  • a computer-implemented method comprising: executing a first user interface having a first button object and a second button object, the first button object being associated with a value, a label, and a digital image; simulating human interaction with the first button object; executing a second user interface as a result of simulating the human interaction with the second button object; identifying that the second user interface includes an occurrence of the label; generating integration code based on locations of: the label and the digital image in the first user interface; and the occurrence of the value in the second user interface; and as a result of execution of the integration code by a client device, causing the client device to display data obtained based on the locations.
  • simulating the human interaction includes emulating a click event or a touch event directed to a respective object.
  • a system comprising: one or more processors; and memory including executable instructions that, if executed by the one or more processors, cause the system to: execute a first interface having a first control object and a second control object, the first control object being associated with a value, a label, and a digital image; engage the first control object; engage the second control object to execute a second interface; identify that the second interface includes an occurrence of the label; determine locations of: the label and the digital image in the first interface; and the occurrence of the value in the second interface; and generate integration code based on the locations; and provide the integration code to a device to cause, as a result of execution of the integration code by the device, the device to display data obtained based on the locations.
  • executable instructions further include instructions that further cause the system to: provide a software application to the device; receive, as a result of execution of the software application by the device, receive a request from the device for the integration code; and provide the integration code in response to the request.
  • executable instructions further include instructions that further cause the system to: receive, further as the result of execution of the software application by the device, an additional request from the device for additional integration code for a third interface; obtain the additional integration code that corresponds to the third interface; and provide the additional integration code in response to the additional request.
  • executable instructions further include instructions that further cause the system to: provide the software application to additional device; receive, as a result of execution of the software application by the additional device, an additional request from the additional device for the integration code; and provide the integration code to the additional device in response to the additional request.
  • the software application further includes software program code that further causes the device to obtain the at least a portion of the data from the local storage of the device.
  • a non-transitory computer-readable storage medium storing thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least: execute a first user interface having a first control and a second control, the first control being associated with a value, a label, and a digital image; engage the second control to execute a second user interface; engage the first control; refresh the second user interface; identify that the second user interface includes an occurrence of the label; generate integration code based on locations of: the label and the digital image in the first user interface; and the occurrence of the value in the second user interface; and provide the integration code to the computing device to cause, as a result of execution of the integration code by the computing device, the computing device to display data obtained, based on the locations, from one or more user interfaces.
  • executable instructions further include instructions that further cause the computer system to: identify a set of interface objects in the first user interface; provide the set of interface object as input to a set of machine learning classifiers to obtain a set of classifications; and determine, based on the set of classifications, first control and the second control.
  • a computer-implemented method comprising: executing a first user interface having a first button object and a second button object, the first button object being associated with a first value; simulating human interaction with the first button object a first time; simulating the human interaction with the second button object to execute a second user interface in a first state; simulating the human interaction with the first button object a second time; refreshing the second user interface to produce the second user interface in a second state; identifying, between a second value in the second user interface in the second state and another second value in the second user interface in the first state, a difference that has a numeric amount of one; identifying, based on the first value, a third value in the second user interface that is associated with the second value; obtaining locations of the second value and the third value in the second user interface; generating integration code based on metadata that indicates the locations; and providing the integration code to a client device to cause, as a result of execution of the integration code by the client device, the client device to: obtain data from
  • simulating the human interaction includes simulating a click event or a touch event directed to the first button object or the second button object.
  • a system comprising: one or more processors; and memory including executable instructions that, if executed by the one or more processors, cause the system to: execute a first interface having a first control object and a second control object, the first control object being associated with a first value; engage the first control object a first time; engage the second control object to execute a second interface in a first state; engage the first control object a second time; re-execute the second interface to produce the second interface in a second state; identify, between a second value in the second interface in the second state and another second value in the second interface in the first state, a difference that has a numeric amount of one; identify, based on the first value, a third value in the second interface that is associated with the second value; obtain locations of the second value and the third value in the second interface; generate integration code based on metadata that indicates the locations; and provide the integration code to a device to cause, as a result of execution of the integration code by the device, the device to: obtain data from the locations in at
  • executable instructions further include instructions that further cause the system to: identify a set of objects in the first interface, the set of objects including the first control object, the second control object, and the first value; provide the set of objects to at least one machine learning classifier to produce a set of potential associations between the set of objects; and determine, based on the set of potential associations, that the first control object is associated with the first value.
  • a non-transitory computer-readable storage medium storing thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least: execute a first user interface having a first control and a second control, the first control being associated with a first value; engage the first control a first time; engage the second control to execute a second user interface in a first state; engage the first control a second time; refresh the second user interface to produce the second user interface in a second state; identify, between a second value in the second user interface in the second state and another second value in the second user interface in the first state, a difference that has a numeric amount of one; identify, based on the first value, a third value in the second user interface that is associated with the second value; obtain locations of the second value and the third value in the second user interface; generate integration code based on metadata that indicates the locations; and provide the integration code to a computing device to cause, as a result of execution of the integration code by the computing device, the computing device to:
  • the executable instructions further include instructions that further cause the computer system to: determine a first set of objects in the second user interface in the first state, the first set of objects having a first set of attribute values; and determine a second set of objects in the second user interface in the second state, the second set of objects having a second set of attribute values; and the executable instructions that cause the computer system to identify the difference that has the numeric amount of one further include instructions that further cause the computer system to determine that an attribute value of an object in the second set of objects is incremented by one from an attribute value of a corresponding object in the first set of objects.
  • the executable instructions further include instructions that further cause the computer system to provide, to the computing device, an application that includes at least one interface object; and the computing device, as a result of execution of the application, further executes the integration code in response to interaction by a user with the at least one interface object.
  • the conjunctive phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: ⁇ A ⁇ , ⁇ B ⁇ , ⁇ C ⁇ , (A, B ⁇ , (A, C ⁇ , (B, C ⁇ , (A, B, C ⁇ .
  • ⁇ A ⁇ , ⁇ B ⁇ , ⁇ C ⁇ A, B ⁇ , (A, C ⁇ , (B, C ⁇ , (A, B, C ⁇ .
  • Processes described can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer software programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof.
  • the code can be stored on a computer-readable storage medium, for example, in the form of a computer software program comprising a plurality of instructions executable by one or more processors.
  • the computer-readable storage medium is non-transitory.
EP20811101.3A 2019-11-11 2020-11-10 Dynamische lokalisierung und extraktion eines benutzerschnittstellenelementzustands in einer benutzerschnittstelle, die von einem ereignis einer anderen benutzerschnittstelle abhängt Pending EP4058888A1 (de)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US16/680,408 US20210141498A1 (en) 2019-11-11 2019-11-11 Unsupervised location and extraction of quantity and unit value elements in a user interface
US16/680,403 US11366645B2 (en) 2019-11-11 2019-11-11 Dynamic identification of user interface elements through unsupervised exploration
US16/680,392 US11379092B2 (en) 2019-11-11 2019-11-11 Dynamic location and extraction of a user interface element state in a user interface that is dependent on an event occurrence in a different user interface
US16/680,406 US11442749B2 (en) 2019-11-11 2019-11-11 Location and extraction of item elements in a user interface
US16/680,396 US11726752B2 (en) 2019-11-11 2019-11-11 Unsupervised location and extraction of option elements in a user interface
PCT/IB2020/060585 WO2021094925A1 (en) 2019-11-11 2020-11-10 Dynamic location and extraction of a user interface element state in a user interface that is dependent on an event occurrence in a different user interface

Publications (1)

Publication Number Publication Date
EP4058888A1 true EP4058888A1 (de) 2022-09-21

Family

ID=73498090

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20811101.3A Pending EP4058888A1 (de) 2019-11-11 2020-11-10 Dynamische lokalisierung und extraktion eines benutzerschnittstellenelementzustands in einer benutzerschnittstelle, die von einem ereignis einer anderen benutzerschnittstelle abhängt

Country Status (4)

Country Link
EP (1) EP4058888A1 (de)
AU (1) AU2020383880A1 (de)
CA (1) CA3157634A1 (de)
WO (1) WO2021094925A1 (de)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7039875B2 (en) * 2000-11-30 2006-05-02 Lucent Technologies Inc. Computer user interfaces that are generated as needed
US8533664B2 (en) * 2009-01-07 2013-09-10 Oracle International Corporation Method and system to automatically generate GUI object addressing queries
US10845949B2 (en) * 2015-09-28 2020-11-24 Oath Inc. Continuity of experience card for index
US10684761B2 (en) * 2017-10-13 2020-06-16 International Business Machines Corporation Graphical user interface creation using highly rated GUI elements

Also Published As

Publication number Publication date
AU2020383880A1 (en) 2022-06-23
CA3157634A1 (en) 2021-05-20
WO2021094925A1 (en) 2021-05-20

Similar Documents

Publication Publication Date Title
US11379092B2 (en) Dynamic location and extraction of a user interface element state in a user interface that is dependent on an event occurrence in a different user interface
US11442749B2 (en) Location and extraction of item elements in a user interface
US11366645B2 (en) Dynamic identification of user interface elements through unsupervised exploration
US20220043810A1 (en) Reinforcement learning techniques to improve searching and/or to conserve computational and network resources
US9460458B1 (en) Methods and system of associating reviewable attributes with items
US11550602B2 (en) Real-time interface classification in an application
US11726752B2 (en) Unsupervised location and extraction of option elements in a user interface
US20150046780A1 (en) Anchoring for content synchronization
US10290040B1 (en) Discovering cross-category latent features
US20180137589A1 (en) Contextual personalized list of recommended courses
US11386356B2 (en) Method of training a learning system to classify interfaces
US10824627B2 (en) Method and system for determining rank positions of non-native items by a ranking system
US9767204B1 (en) Category predictions identifying a search frequency
US20210141498A1 (en) Unsupervised location and extraction of quantity and unit value elements in a user interface
US10387934B1 (en) Method medium and system for category prediction for a changed shopping mission
US20220050560A1 (en) Extraction and restoration of option selections in a user interface
US20220366264A1 (en) Procedurally generating realistic interfaces using machine learning techniques
US20230306071A1 (en) Training web-element predictors using negative-example sampling
EP4058888A1 (de) Dynamische lokalisierung und extraktion eines benutzerschnittstellenelementzustands in einer benutzerschnittstelle, die von einem ereignis einer anderen benutzerschnittstelle abhängt
WO2023148618A1 (en) Dynamic labeling of functionally equivalent neighboring nodes in an object model tree
US11409546B2 (en) Interface classification system
CA3162860A1 (en) Interface classification system
US20230097572A1 (en) Optimizing questions to retain engagement
US20220164194A1 (en) Unified semantic model of user intentions

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220504

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)