CN113168355A - Automating processes running in a first session via a robotic process automation robot running in a second session - Google Patents

Abstract

Automating processes running in a first session via a Robot Process Automation (RPA) robot(s) running in a second session is disclosed. In some aspects, the form is displayed in a user session, but one or more attended RPA robots that acquire and/or interact with data for an application in a first session run in one or more other sessions. In this manner, while the RPA robot(s) are running, the operation of the RPA robot(s) may not prevent the user from using other applications or instances, but the data modifications made or facilitated by the RPA robot(s) may be visible to the user in the first session window.

Description

Automating processes running in a first session via a robotic process automation robot running in a second session

Cross Reference to Related Applications

This application claims priority to U.S. non-provisional patent application No. 16/924,910 filed on 9/7/2020 and the benefit of U.S. non-provisional patent application No. 16/925,544 filed on 10/7/2020 as a continuation thereof. The subject matter of these earlier filed applications is incorporated herein by reference in its entirety.

Technical Field

The present application relates generally to Robot Process Automation (RPA), and more particularly to automating processes running in a first session via RPA robot(s) running in a second session.

Background

For example, a manned automated RPA robot typically runs on a computing system operated by a user in the same session as the user is in. For example, under the command of a user, the RPA robot may work with the user to accomplish certain tasks. However, in a manned automation scenario, the RPA robot may "take over" the user's computing system. While the bot is interacting with the computing system, the user may wish to perform other activities, but the user is prevented from performing other activities. In other words, the robot controls applications (e.g., simulating mouse clicks and keyboard inputs) through a User Interface (UI) in the same manner as a user.

Various techniques exist for creating a complete or partial copy of an operating system or application running thereon. Simulators have been proposed for decades and may provide developers with the ability to test and debug applications. For example, an emulator may provide a developer with the ability to test and debug mobile applications that use operating systems that do not support direct execution of development tools.

And

provide a test and debug executable from a development machine

Or

Emulators of applications, thisBecause developer tools cannot run natively on those mobile operating systems.

Simulators allow developers to host windows on their local machines that let the developers test and debug the behavior of applications that are difficult or impossible to execute on the development machines. For example, to test and debug application behavior that responds to these events, the simulator allows the developer to click on a button to rotate the simulator, which informs the application running inside the simulator that the device has rotated. Another common example is multi-touch. Many developer machines do not support touch, so the simulator can let the developer test and debug how the application responds to multiple touch points.

And

the emulator also provides simulation capabilities. In addition to this, the present invention is,

simulators are provided for their Universal Windows Platform (UWP) applications.

The virtual machine hosts a second operating system on the machine and may be opened and monitored through the window. This runs a completely different operating system and shares hardware with the host. The "guest" machine must install its own copy of the application and cannot share common resources or files with the user machine.

A container is conceptually a hybrid form of a virtual machine. All applications that need to be executed are encapsulated in an immutable package that is executed directly on the host operating system. The package is not a complete copy of another operating system, but the package does not share or have access to any applications or resources on the host by default. Thus, from a user experience perspective,

containers are perceptually similar to virtual machines, but technically, these containers are not executed on completely separate operating systems.

However, conventional emulators, Virtual Machines (VMs), and hybrid VMs (e.g., Operating System (OS) level virtualization are provided

Container) does not address the problems that arise with manned automated robots operating on the same computing system as the user. Thus, the user essentially becomes a bystander of his own computing system who monitors the operation of the robot but cannot interact with other applications on the machine that require user interaction. Accordingly, an improved method may be beneficial.

Disclosure of Invention

Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, appreciated, or solved by current RPA techniques. For example, some embodiments of the invention relate to automating processes running in a first session via RPA robot(s) running in a second session.

In an embodiment, a computer program is embodied on a non-transitory computer readable medium. The computer program is configured to cause at least one processor to: performing an inter-process communication (IPC) facilitator and an RPA driver in a first session; and executing the RPA robot in the second session. The computer program is further configured to cause the at least one processor to: receiving, by the IPC facilitator, one or more messages from the RPA robot via IPC; and controlling, by the IPC facilitator, the RPA driver to interact with the application or application object running in the first session based on the one or more messages received from the RPA robot.

In another embodiment, a computer-implemented method includes: sending, by the RPA robot running in the second session, one or more messages to the IPC facilitator running in the first session via IPC; and receiving, by the IPC facilitator, one or more messages from the RPA robot via IPC. The computer-implemented method further comprises: controlling, by the IPC facilitator, the RPA driver running in the first session to interact with the application or application object also running in the first session based on the one or more messages received from the RPA robot.

In yet another embodiment, a computer-implemented method includes: one or more messages are received by an IPC facilitator running in a first session from an RPA robot running in a second session via IPC. The computer-implemented method further comprises: controlling, by the IPC facilitator, the RPA driver running in the first session to interact with the application or application object also running in the first session based on the one or more messages received from the RPA robot.

Drawings

In order that the advantages of certain embodiments of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. While the present invention will be described and illustrated with additional specificity and detail through the use of the accompanying drawings in which:

fig. 1 is an architecture diagram illustrating a Robotic Process Automation (RPA) system according to an embodiment of the present invention.

Fig. 2 is an architecture diagram illustrating a deployed RPA system in accordance with an embodiment of the present invention.

FIG. 3 is an architecture diagram illustrating the relationship between designer, activity and drivers according to an embodiment of the invention.

Fig. 4 is an architecture diagram illustrating an RPA system according to an embodiment of the present invention.

Fig. 5 is an architecture diagram illustrating a computing system configured to facilitate inter-session automation for an RPA robot, in accordance with an embodiment of the present invention.

6A-G illustrate an example of completing a form in a user session using an RPA robot, IPC facilitator, and RPA driver running in a robot session, according to an embodiment of the present invention.

7A-G illustrate examples of completing a form in a user session via direct variable modification using an RPA robot running in the robot session, in accordance with an embodiment of the present invention.

FIG. 8 is a flow diagram illustrating a process of automating processes running in a user session via an RPA robot, IPC facilitator, and RPA driver running in a robot session, according to an embodiment of the invention.

Fig. 9 is a flow diagram illustrating a process of automating processes running in a user session via RPA robots running in the robot session via direct variable modification, according to an embodiment of the invention.

FIG. 10 illustrates an example of a plurality of client session RPA bots interacting with a master session application in accordance with an embodiment of the present invention.

FIG. 11 is a flow diagram illustrating a process for running a manned automated RPA robot in a client session in accordance with an embodiment of the present invention.

FIG. 12 is a flow diagram illustrating the execution of a multi-robot collective workflow between a master session robot M1, a client session application A1, and a pair of client session robots C1 and C2 in accordance with an embodiment of the present invention.

Fig. 13 is a flowchart illustrating parallel execution of a master session RPA robot and a client session RPA robot in accordance with an embodiment of the present invention.

FIG. 14 is a flow diagram illustrating execution of a client session RPA robot based on a trigger for a master session application in accordance with an embodiment of the present invention.

FIG. 15 is a flow diagram illustrating a process for performing inter-session automation in accordance with an embodiment of the present invention.

Unless otherwise indicated, like reference numerals designate corresponding features throughout the drawings.

Detailed Description

Some embodiments involve automating processes running in a first session via RPA robot(s) running in a second session. For example, forms (e.g., Web pages, email applications (e.g., the like)

) Spreadsheet application (e.g. for the same purpose)

) Customer Relationship Management (CRM) system applications (e.g., for example)

) An Enterprise Resource Management (ERM) system, a supply chain management system, a customized computer application, a mobile application, another application with fillable fields, etc.) may be displayed in the first session. In some embodiments, the first session may be a user session (also referred to herein as a master session or parent session). One or more RPA bots that acquire and/or interact with data of an application in a first session may run in one or more other sessions (also referred to herein as robotic sessions, second sessions, or auxiliary sessions). In some embodiments, the robot session may be a sub-session of the first session.

In some embodiments, the RPA bot running in a client session sends a message to the RPA driver running in the master session via inter-process communication (IPC). Based on IPC communication(s) from the RPA robot, the RPA driver may enable various Operating System (OS) and/or Application Programming Interface (API) level interactions with the computing system. For example, the actuation may move a mouse, click a button, interact with a menu, enter text in a field, open or close a window, move a window, and/or resize a window, or perform any other suitable interaction, without departing from the scope of the invention.

To implement this functionality, the driver may include or be controlled by or invoked by the IPC facilitator. In some embodiments, the IPC facilitator may be a separate application or process, or may be a driven sub-process. The IPC facilitator can listen for IPC messages (e.g., listening triggers) from the RPA robots in a client session, send and receive communications to and from the RPA robots, monitor RPA robot execution status, combinations thereof, and the like. The IPC facilitator can also cause the driver to implement the desired functionality in the parent session based on the IPC communication(s) from the RPA robot. In some embodiments, the IPC facilitator may be configured with a script for a given RPA robot. In some embodiments, the IPC facilitator may be custom designed for the functionality to be performed in the parent session.

In some embodiments, the RPA bot in a client session may directly modify variables stored in memory on the computing system. An application running in the parent session that accesses these variables may then update the displayed values when the display refreshes or the application otherwise reloads its UI. For example, it may be modified directly by the RPA robot running in the client session

A variable of the spreadsheet, which also changes the value displayed in the parent session.

In some embodiments, the operation of the RPA bot(s) may not prevent the user from using other applications or instances while the RPA bot is running, but the data modifications by the RPA bot(s) may be visible to the user when the application display refreshes in the first user session window. However, in some embodiments, the RPA bot(s) may interact with an application that does not have a user interface.

In some embodiments, the process may operate as follows. The user may click a button or otherwise cause the RPA bot to start in a sub-session of a user session (e.g., a parent session or a master session that is started when the user turns on their computing system). In some embodiments, for example, the robotic session may already be in operation or may be initiated by the user separately from the robotic tray. The user may have the bot execute its workflow in a sub-session, and the bot may then interact with one or more applications in the main session by communicating with an IPC facilitator running in a parent session. The IPC facilitator may then cause the RPA driver to implement the desired functionality in the parent session.

In some embodiments, if at least one application to which the workflow belongs has not been launched, the RPA robot running in the robot session may pause operation and provide a message, and the IPC facilitator may launch the application so that the robot may continue operation. Once the robot begins to perform its workflow, changes to the displayed data in the application may be visible in the main session window when such changes are made via the IPC facilitator and RPA driver. For example, the data may be visualized as the driver populates the data based on the data provided by the RPA robot in the client session (e.g., via IPC that saves the data into a flat file or database, etc.).

By way of non-limiting example, in some embodiments, may be

A shortcut is created that causes the RPA robot to start in a client session or to interact with a running RPA robot in a client session. When the user clicks on a shortcut, the RPA robot starts to perform the RPA process as designed in its RPA workflow. Based on the results of its execution (e.g., obtaining information from a database, invoking an Artificial Intelligence (AI)/Machine Learning (ML) model and obtaining results, obtaining information from multiple legacy systems, etc.), the RPA robot then interacts with an IPC facilitator, which then interacts with the running IPC facilitator in a parent session

Application instance interactions. This may result, for example, in the RPA driver performing the interaction, being operational

Some of the displayed information in the application instance changes and is visible to the user.

In some embodiments, an RPA bot running in a client session may cause an application in a parent session to open via an IPC facilitator. The IPC facilitator can then access data associated with the application via the RPA driver and make additions, changes, and/or deletions. In this way, even if the RPA bot is not running in the parent session, the RPA bot can still cause interactions in the parent session application in a manner that appears similar to the RPA bot running in the parent session itself, but differs functionally.

In some embodiments, the user may have the bot execute a workflow that visits a website and collects some information in a client session. For example, the RPA robot may then provide this information to an IPC facilitator, which then causes the RPA driver to enter the collected information in a spreadsheet that is visible in the master session. In some embodiments, applications (such as

) Open in the master session. The user then runs the automation in a client session that reads the current client ID, and enters the website (e.g., the user enters the website)

) And collect information related to client interactions with the website. However, the manned RPA robot may interact with any suitable application(s) via the RPA facilitator and/or may obtain data from any suitable source (e.g., a database, another application, etc.) without departing from the scope of the present invention.

Applications of some embodiments include, but are not limited to, emulators, simulators, VMs, and hybrid VMs (e.g., such as those providing OS-level virtualization

A container). Some embodiments create and host one or more robotic sessions in the form of a window that includes a UI of an application controlled by an attended automation process. In some embodiments, only the interfaces of the applications with which the robot(s) interact are shown. In some embodiments, the client session window is not shown at all. As used herein, a "window" may apply to a window representing a UI shown within a main UI, a second display of a computing system, without departing from the scope of the inventionThe second screen of the bot, the virtual desktop, the isolated environment (i.e., the window (referred to as the "host"), etc., which draws the UI of all applications launched inside the environment (referred to as the "child") and runs them in the context of the host session.

In some embodiments, any desired number of sessions for any number of robots may be created and used without departing from the scope of the present invention. For example, a user may operate in a first session, a first robot may operate in a second session, a second robot may operate in a third session, and so on. In some embodiments, multiple robots may operate in a single session, potentially interacting in turn with one or more public applications via the RPA facilitator.

For example, can be via

The terminal service sub-session implements the functionality for creating a session, the

The terminal services sub-session may create the session back into the user's own machine without the user logging off. The newly created session appears as a child window and contains and launches the applications that exist in the user session. In other words, the separation between the user and the robot occurs at the UI level. For example, if a file is deleted, this may occur in all sessions running on the computing system.

Certain embodiments may be deployed for Robotic Process Automation (RPA). Fig. 1 is an architectural diagram illustrating an RPA system 100 in accordance with an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows a developer to design and implement a workflow. Designer 110 may provide solutions for application integration and automation of third party applications, administrative Information Technology (IT) tasks, and business IT processes. The designer 110 can support the development of an automation project, which is a graphical representation of a business process. In short, the designer 110 supports the development and deployment of workflows and robots.

Automation projects enable automation of rule-based processes by providing developers control over the execution order and relationships between a set of custom steps (defined herein as "activities") developed in a workflow. One commercial example of an embodiment of designer 110 is the UiPadh Studio^TM. Each activity may include an action, such as clicking a button, reading a file, writing a log panel, and so forth. In some embodiments, workflows can be nested or embedded.

Some types of workflows may include, but are not limited to, a sequence, a flow diagram, a FSM, and/or a global exception handler. The sequence may be particularly suited to a linear process, enabling flow from one activity to another without cluttering the workflow. The flow diagrams may be particularly suited to more complex business logic, enabling the integration of decisions and active joining in a more diverse manner by multiple branch logic operators. The FSM may be particularly suited for larger workflows. The FSM may use a limited number of states in its execution, which are triggered by conditions (i.e., transitions) or activities. The global exception handler may be particularly suited to determine workflow behavior when an execution error is encountered and particularly suited to the debugging process.

Once the workflow is developed in the designer 110, execution of the business process is orchestrated by a director (conductor)120 that orchestrates one or more robots 130 that execute the workflow developed in the designer 110. One commercial example of an embodiment of director 120 is the UiPath Orchester^TM. Director 120 supports the management of the creation, monitoring, and deployment of resources in an environment. With third party solutions and applications, the director 120 may act as one of an integration point, or as an aggregation point.

The director 120 may manage a fleet of robots 130 to connect and execute the robots 130 from a centralized point. The types of robots 130 that may be managed include, but are not limited to: manned 132, unmanned 134, development robots (similar to unmanned 134 but for development and testing purposes), and non-production robots (similar to manned 132 but for development and testing purposes). The manned robot 132 may be triggered by a user event or arranged to occur automatically and operate on the same computing system as a human. The manned robot 132 may be used with the director 120 to centralize process deployment and recording media. The manned robot 132 may assist a human user in completing various tasks and may be triggered by user events. In some embodiments, processes cannot be started from the director 120 on this type of robot and/or run under the lock screen. In some embodiments, the manned robot 132 may only be turned on from the robot pallet or upon command prompts. In some embodiments, the manned robot 132 should operate under human supervision.

The unattended robot 134 operates in an unattended manner in a virtual environment and can automate many processes. The unattended robot 134 may be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types may be run from the designer 110. Both manned and unmanned robots can automate various systems and applications including, but not limited to, mainframe, Web applications, VMs, enterprise applications (e.g., by a mainframe, Web application, VM, etc.)

Etc.) as well as computing system applications (e.g., desktop and notebook applications, mobile device applications, wearable computer applications, etc.).

Director 120 can have various capabilities including, but not limited to, provisioning, deployment, versioning, configuration, queuing, monitoring, logging, andand/or provide interconnectivity. Provisioning may include creating and maintaining a connection between the bot 130 and the director 120 (e.g., a Web application). Development may include ensuring that a package version is properly delivered to the assigned robot 130 for execution. In some embodiments, versioning may include the management of unique instances of a certain process or configuration. The configuration may include maintenance and delivery robot environment and process configuration. Queuing may include providing management of queues and queue entries. Monitoring may include tracking robot identification data and maintaining user privileges. Logging can include storing and indexing the log to a database (e.g., an SQL database) and/or another storage mechanism (e.g., a database with SQL data)

Which provides the ability to store and quickly query larger data sets). The director 120 can provide interconnectivity by acting as a centralized point of communication for third party solutions and/or applications.

The robot 130 is an executing agent that runs a workflow built into the designer 110. One commercial example of some embodiments of the robot(s) 130 is the UiPad Robots^TM. In some embodiments, the robot 130 installs Microsoft by default

Services managed by a Service Control Manager (SCM). As a result, such robots 130 can open interactive under local system accounts

A conversation and has

The rights of the service.

In some embodiments, the robot 130 may be installed in user mode. For such a robot 130, this means that it has the same rights as the user who has installed the given robot 130. This feature can also be used with High Density (HD) robots, which ensure full utilization of each machine at its maximum potential. In some embodiments, any type of robot 130 may be configured in an HD environment.

In some embodiments, the robot 130 is divided into several components that are each dedicated to a particular automation task. In some embodiments, the robotic components include, but are not limited to, SCM managed robotic services, user mode robotic services, actuators, agents, and command lines. SCM managed robot service management and monitoring

A session, and acts as a proxy between the director 120 and the executing host (i.e., the computing system on which the robot 130 is executed). These services are trusted by the robot 130 and manage credentials of the robot 130. The console application is launched by the SCM under the local system.

In some embodiments, user mode robotic service management and monitoring

A session and acts as a delegate between director 120 and the executing host. The user-mode robot service may be trusted by the robot 130 and manage credentials of the robot 130. In case that the robot service managed by the SCM is not installed,

the application may be automatically launched.

The actuator can be at

Running a given job under a session (i.e., an executor may execute a workflow; an executor may know the point per inch (DPI) setting for each monitor

Presentation Foundation (WPF) application, which displays available jobs in a system tray window. The proxy may be a client of the service. The agent may request to turn on or offStop work and change settings. The command line is a client of the service. A command line is a console application that can request to start a job and wait for its output.

Separating the components of the bot 130 as explained above generally helps developers, supporting users, and computing systems to more easily run, identify, and track what each component is executing. Special behaviors may be configured by component in this manner, such as setting different firewall rules for enforcers and services. In some embodiments, the actuator may always know the DPI settings of each monitor. As a result, the workflow may be executed at any DPI regardless of the configuration of the computing system that created the workflow. In some embodiments, the project from designer 110 may also be independent of browser zoom level. In some embodiments, DPI may be disabled for applications where DPI is unknown or intentionally marked as unknown.

Fig. 2 is an architectural diagram illustrating a deployed RPA system 200 in accordance with an embodiment of the present invention. In some embodiments, RPA system 200 may be, or may be a part of, RPA system 100 of fig. 1. It should be noted that the client-side, server-side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robotic application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, designer 216 may not run on computing system 210. The actuator 212 is running the process. As shown in fig. 2, several business items may be running simultaneously. In this embodiment, agent 214 (e.g., a proxy)

Service) is a single point of contact for all actuators 212. All messages in this embodiment are logged into director 230, which further processes the messages via database server 240, indexer server 250, or both. As discussed above with respect to fig. 1, the implement 212 may be a robotic assembly.

In some embodiments, the robot represents a relationship between a machine name and a user nameAnd (6) associating. The robot may manage multiple actuators simultaneously. In a computing system that supports multiple interactive sessions running simultaneously (e.g., in a web server

Server 2012), multiple robots may be running simultaneously, each using a separate user name

In a conversation. This is referred to as HD robot above.

The agent 214 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still functioning) and downloading the required version of the package to be executed. In some embodiments, communication between the agent 214 and director 230 is always initiated by the agent 214. In a notification scenario, agent 214 may open a WebSocket channel that is later used by director 230 to send commands (e.g., turn on, stop, etc.) to the robot.

On the server side, there are a presentation layer (Web application 232, open data protocol (OData) representational state transfer (REST) Application Programming Interface (API) endpoint 234 and notification and monitoring 236), a service layer (API implementation/business logic 238), and a persistence layer (database server 240 and indexer server 250). Director 230 includes a Web application 232, an OData REST API endpoint 234, notification and monitoring 236, and API implementation/business logic 238. In some embodiments, most of the actions performed by the user in the interface of director 230 (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, opening a job on the robot, adding/removing data in a queue, scheduling a job to run in an unattended manner, etc., without departing from the scope of the present invention. The Web application 232 is the visual layer of the server platform. In this embodiment, the Web application 232 uses hypertext markup language (HTML) and javascript (js). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. In this embodiment, a user interacts with a Web page from a Web application 232 via the browser 220 to perform various actions to control the director 230. For example, a user may create a group of robots, assign a package to the robots, analyze a log of each robot and/or each process, start and stop the robots, and so on.

In addition to the Web application 232, the director 230 also includes a service layer that exposes an OData RESTAPI endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. RESTAPI is consumed by both Web application 232 and proxy 214. In this embodiment, agent 214 is a supervisor of one or more robots on a client computer.

In this embodiment, RESTAPI overrides configuration, logging, monitoring, and queuing functionality. In some embodiments, configuration endpoints may be used to define and configure application users, permissions, bots, assets, publications, and environments. Logging REST endpoints may be used to log different information such as, for example, errors, explicit messages sent by the robot, and other environment-specific information. If a start job command is used in director 230, the deployment REST endpoint can be used by the robot to query the version of the package that should be executed. Queuing REST endpoints may be responsible for queue and queue entry management, such as adding data to the queue, obtaining transactions from the queue, setting the state of the transaction, and the like.

Monitoring the REST endpoint may monitor the Web application 232 and the agent 214. The notification and monitoring API236 may be a REST endpoint for registering the proxy 214, delivering configuration settings to the proxy 214, and for sending/receiving notifications from servers and the proxy 214. In some embodiments, the notification and monitoring API236 may also communicate using WebSocket.

In this embodiment, the persistence layer includes a pair of servers — a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, database server 240 stores the configuration of robots, groups of robots, associated processes, users, roles, schedules, and the like. In some embodiments, this information is managed by the Web application 232. Database server 240 may manage queues and queue entries. In some embodiments, the database server 240 may store messages logged by the robot (in addition to the indexer server 250 or in place of the indexer server 250).

An optional indexer server 250 in some embodiments stores and indexes information recorded by the robot log. In certain embodiments, the indexer server 250 may be disabled by configuration settings. In some embodiments, the indexer server 250 uses

Which is an open source project full-text search engine. Messages logged by the robot (e.g., using activities such as log messages or write lines) can be sent through the logging REST endpoint(s) to the indexer server 250 where they are indexed for future use in the indexer server 250.

Figure 3 is an architecture diagram illustrating a relationship 300 between a designer 310, activities 320, 330, and drivers 340, according to an embodiment of the invention. From the above, the developer uses designer 310 to develop the workflow performed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as Computer Vision (CV). Some CV activities related to such components may include, but are not limited to, clicking, typing, retrieving text, hovering, element presence, refreshing a scope, highlighting, and so forth. In some embodiments, the click identifies the element using, for example, CV, Optical Character Recognition (OCR), fuzzy text matching, and multi-anchor (multi-anchor), and clicks on the element. Typing may use the above to identify an element and a type in the element. Retrieving text may identify a location of a particular text and scan the location using OCR. The hover may identify the element and hover over the element. Element presence may use the techniques described above to check whether an element is present on the screen. In some embodiments, there may be hundreds or even thousands of activities that may be implemented in designer 310. However, any number and/or type of activities may be available without departing from the scope of the present invention.

The UI automation activity 330 is a subset of a particular lower level activity (e.g., CV activity) written in lower level code and facilitates interaction with the screen. The UI automation activity 330 supports these interactions via drivers 340 that allow the robot to interact with the desired software. For example, the drivers 340 may include an OS driver 342, a browser driver 344, a VM driver 346, an enterprise application driver 348, and the like.

The driver 340 may interact with the OS at a low level to find hooks, monitor keys, etc. The driving may be assisted with

And the like. For example, the "click" activity plays the same role in these different applications via the driver 340.

Fig. 4 is an architecture diagram illustrating an RPA system 400 in accordance with an embodiment of the present invention. In some embodiments, RPA system 400 may be or include RPA systems 100 and/or 200 of fig. 1 and/or 2. RPA system 400 includes a plurality of client computing systems 410 running robots. The computing system 410 is capable of communicating with the director computing system 420 via a Web application running thereon. Director computing system 420 can in turn communicate with database server 430 and optional indexer server 440.

With respect to fig. 1 and 3, it should be noted that although a Web application is used in these embodiments, any suitable client/server software may be used without departing from the scope of the present invention. For example, the director may run a server-side application that communicates with a non-web-based client software application on the client computing system.

Fig. 5 is an architecture diagram illustrating a computing system 500 configured to facilitate inter-session automation for an RPA robot in accordance with an embodiment of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems depicted and/or described herein. Computing system 500 includes a bus 505 or other communication mechanism for communicating information, and processor(s) 510 coupled to bus 505 for processing information. Processor(s) 510 may be any type of general or special purpose processor including a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Graphics Processing Unit (GPU), multiple instances thereof, and/or any combination thereof. Processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of the processor(s) 510 may be a neuromorphic circuit that includes a processing element that mimics a biological nerve. In some embodiments, the neuromorphic circuit may not require the typical components of a von neumann computing architecture.

Computing system 500 also includes memory 515 for storing information and instructions to be executed by processor(s) 510. The memory 515 may be comprised of any combination of Random Access Memory (RAM), Read Only Memory (ROM), flash memory, cache, static storage device (such as a magnetic or optical disk), or any other type of non-transitory computer-readable medium or combination thereof. Non-transitory computer-readable media can be any available media that can be accessed by the processor(s) 510 and can include volatile media, non-volatile media, or both. The media may also be removable, non-removable, or both.

Additionally, computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via a wireless and/or wired connection. In some embodiments, communications device 520 may be configured to use Frequency Division Multiple Access (FDMA), single carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, wideband CDMA (W-CDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), High Speed Packet Access (HSPA), Long Term Evolution (LTE), LTE-advanced (LTE-a), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, home node b (hnb), bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Near Field Communications (NFC), fifth generation (5G), and/or the like without departing from the scope of the invention, A New Radio (NR), any combination thereof, and/or any other existing or future implemented communication standard and/or protocol. In some embodiments, the communication device 520 may include one or more antennas that are singular, array, phased, switched, beam formed, beam steered, combinations thereof, and/or any other antenna configuration without departing from the scope of the present invention.

Processor(s) 510 is further coupled via bus 505 to a display 525, such as a plasma display, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, a Field Emission Display (FED), an Organic Light Emitting Diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a liquid crystal display, a light emitting diode (OLED) display, a Field Emission Display (FED), an Organic Light Emitting Diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a liquid crystal display, and a display,

A display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (tactile) display, three-dimensional (3D) touch display, multi-input touch display, multi-touch display, and the like, using resistive, capacitive, Surface Acoustic Wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, and the like. Any suitable display device and tactile I/O may be used without departing from the scope of the invention.

A keyboard 530 and a cursor control device 535, such as a computer mouse, touchpad, and the like, are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and a user may interact with the device only through the display 525 and/or a touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In some embodiments, there is no physical input device and/or display. For example, a user may interact with computing system 500 remotely via another computing system with which the user communicates, or computing system 500 may operate autonomously.

The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 of the computing system 500. The modules also include an inter-session automation module 545 configured to perform all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functionality modules 550 that include additional functionality.

Those skilled in the art will appreciate that a "system" may be implemented as a server, an embedded computing system, a personal computer, a console, a Personal Digital Assistant (PDA), a cell phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices, without departing from the scope of the invention. The presentation of the above described functions as being performed by a "system" is not intended to limit the scope of the invention in any way, but rather is intended to provide one example of many embodiments of the invention. Indeed, the methods, systems, and apparatus disclosed herein may be implemented in localized and distributed forms consistent with computing technologies including cloud computing systems.

It should be noted that some of the system features described in this specification have been presented as modules, in order to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom Very Large Scale Integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, or the like.

Modules may also be at least partially implemented in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized as an object, procedure, or function, for example. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module. Further, the modules may be stored on a computer readable medium, which may be, for example, a hard disk drive, a flash memory device, a RAM, a tape, and/or any other such non-transitory computer readable medium for storing data, without departing from the scope of the present invention.

Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network.

Typically, in manned automation, a human user waits when an RPA robot running in the same session completes its task. However, some embodiments create one or more robot sessions to host and run RPA robots therein. Unlike existing RPA systems, users may benefit from the ability to interact with their computing system while the robot(s) are running automation in the robot session(s). In some embodiments, the user still monitors what the robot is doing and interacts with the robot through the host automation window(s) for the robot session(s). In some embodiments, the user and the robotic session(s) may be running on a remote machine that is controlled by the user's computing system.

However, in some embodiments, the RPA robot does not interact with the application being used by the user via the IPC facilitator. Instead, the RPA robot may interact via the RPA facilitator and RPA driver with an application or other process that is not visible to or otherwise not used by the user. Such robots generally do not directly increase user interaction with the application and may be considered to be unattended robots.

In some embodiments, the RPA robot may be running on a user's computing system and run through a Remote run time (Remote run time) (e.g., via a UiPath Remote run time)^TM) Driving the remote computing system. UiCat Remote Runtime^TMIs to facilitate remote applications or Desktops (such as Citrix virtual apps and Desktops)^TM) And special purpose

Extension (e.g. to

Is/are as follows

Extend or aim at

Of remote desktop

Extensions) of the communication between the components. UiCat Remote Runtime^TMCollecting information related to a target UI element of a remote application and sending the information to a corresponding extension, such that at UI Explorer^TMThe intermediate computer generates a selector.

6A-G illustrate an example of completing a form in a user session using an RPA robot, IPC facilitator, and RPA driver running in a robot session, according to an embodiment of the present invention. In FIG. 6A, a user session window 600 is shown in which a user is able to interact with an application in the UI and no bot is currently executing. The robot tray icon 610 is visible in the lower right portion of the user session window 600. The RPA driver 640 includes an IPC facilitator 642 as a sub-process in this embodiment. However, in certain embodiments, the IPC facilitator 642 may be a separate application or process that communicates with the RPA driver 640 without departing from the scope of the present invention. In such embodiments, the IPC facilitator 642 may communicate with the RPA driver 640 via IPC, via API calls, and the like. Indeed, in some embodiments, IPC may not be used.

In FIG. 6B, the user launches a Web browser and visits an invoice creation Web page 620. In fig. 6C, the user pulls up on the robot tray 612 (e.g., by clicking on the robot tray icon 610) and selects the client session robot option 614 to execute on their computing system. As shown in fig. 6D, after the robot to be executed is selected, a robot session window 630 for robot session is displayed as a sub-window on the screen. The RPA will operate in a robotic session. In this embodiment, a window 632 for the client session bot is automatically launched within the bot session window 630 and includes a button 634 for running a form populator workflow to obtain data for the Web page 620 in a user session.

In some embodiments, the robot session window may not be displayed, and the robot may automatically start, operate, and close without being visible to the user. In some embodiments, the robot may close its session after completing its workflow. In some embodiments, rather than starting from the robot tray, a robot session may be initiated, and the robot may start and operate without using the robot tray 612 (e.g., due to the user clicking a button in an application of the master session).

Turning to FIG. 6E, after the user clicks on button 634, the bot begins to acquire data for populating form fields in Web page 620. When the robot has acquired a portion of the data, it sends the data to the IPC facilitator 642, which then causes the RPA driver 640 to enter the data into the Web browser 620. For example, RPA driver 640 may move a mouse over a given field and cause text to be entered into that field from the data. In fig. 6E, RPA driver 640 has completed the invoice number field and has now moved the mouse to the invoice number field, clicked on the invoice number field, and entered the number 1. In fig. 6E, the insertion symbol appears after the number 1. In some embodiments, RPA facilitator 642 may receive the complete data set from bot 632 before causing RPA driver 640 to fill in the fields of Web page 620.

In accordance with the above, the communication between the robot running in the robot session and the IPC facilitator 642 running in the user session is accomplished using the IPC protocol. IPC protocols may facilitate communication via networks, pipes, Component Object Models (COM), Remote Process Calls (RPCs), sockets, and the like. Suitable session creation mechanisms and IPC protocols may also be used for other supported operating systems. The robot may also send status notifications (e.g., indicating that the robot is starting, running, pausing, etc.), acquired data, error messages, commands, or other communications back to the IPC facilitator 642 via the IPC protocol.

When Web page 620 is not covered or minimized by another window, the associated text from the data acquired by the bot is visible to the user as it is entered by RPA driver 640. In some embodiments, when RPA driver 640 completes the data interaction for the form, the user may interact with other applications and continue to act or even interact with Web page 620 itself, although the user's entry may be overwritten if RPA driver 640 modifies the data for the same field in which the user entered the information.

Turning to FIG. 6F, after the robot has completed its workflow, a message is displayed in window 632. As can be seen in the background, the robot has completed data acquisition for the form fields, and new entries for the form fields are visible in the Web page 620 after being entered by the RPA driver 640. The user may then close the robot session window 630, the robot may automatically close the robot session window 630 (and potentially the robot session itself), or the robot session window 630 may remain open. The user may then submit the completed form. See fig. 6G.

7A-G illustrate examples of completing a form in a user session via direct variable modification using an RPA robot running in the robot session, in accordance with an embodiment of the present invention. In FIG. 7A, a user session window 700 is shown in which a user is able to interact with an application in the UI and no bot is currently executing. The robot tray icon 710 is visible in the lower right portion of the user session window 700.

In FIG. 7B, the user launches a Web browser and visits an invoice creation Web page 720. In fig. 7C, the user pulls up on the robot tray 712 (e.g., by clicking on the robot tray icon 710) and selects the client session robot option 714 for execution on their computing system. As shown in fig. 7D, after the robot to be executed is selected, a robot conversation window 730 for robot conversation appears as a sub-window on the screen. The RPA will operate in a robotic session. In this embodiment, a window 732 for the client session bot is automatically launched within the bot session window 730 and includes a button 734 for running a form populator workflow of the Web page 720 in the user session.

In some embodiments, the robot session window may not be displayed, and the robot may automatically start, operate, and close without being visible to the user. In some embodiments, the robot may close its session after completing its workflow. In some embodiments, rather than starting from the robot tray, a robot session may be initiated, and the robot may start and operate without using the robot tray 712 (e.g., due to the user clicking a button in an application of the master session).

Turning to FIG. 7E, after the user clicks the button 734, the bot begins filling out form fields in the Web page 720 by accessing and interacting with variables for the form fields of the Web page 720. For example, the bot may change a stored value (memory value) for a field stored in RAM, persistent local storage, a database, or any other storage type and/or location without departing from the scope of the present invention. In some embodiments, a variable may be stored as part of memory, the variable being assigned for and used by the Web application associated with Web form 720.

When the Web page 720 is refreshed, the corresponding stored value changed by the bot causes the corresponding field of the Web page 720 to be updated. When the Web page 720 is not covered or minimized by another window, the associated text changed by the bot via the data modification is visible to the user. Unlike the example in fig. 6A-G, where RPA driver 640 manipulates a mouse and enters text via an API level and/or native mechanism, data appears in the form field of Web page 720 without such interaction. In some embodiments, when the bot completes the data interaction for the form, the user may interact with other applications and continue to act or even interact with the Web page 720 itself, although the user's entry may be overwritten if the bot modifies data for the same field in which the user entered information.

Turning to FIG. 7F, after the robot has completed its workflow, a message is displayed in window 732. As can be seen in the background, the bot has completed data modifications for the respective storage variable (memory variable) associated with the form field, and new entries for the form field are visible in the network 720 after the network 720 refreshes. The user may then close the robot session window 730, the robot may automatically close the robot session window 730 (and potentially the robot session itself), or the robot session window 730 may remain open. The user may then submit the completed form. See fig. 7G.

In some embodiments, the robot session may be a sub-session, and the sub-session may be created via a sub-session API of the operating system. In some embodiments, may be used without departing from the scope of the invention

The terminal services sub-session or another sub-session API provided by the operating system to create the second session. A robotic pallet application (e.g., configured to launch the robot (s))

Robot agent desktop) or another application may then be operatingA create process API with appropriate parameters is used in the system to start the robot flow in this sub-session. The robot tray application or other suitable application may then communicate with the robot process using a suitable protocol (e.g., a protocol built on a named pipe).

FIG. 8 is a flow diagram illustrating a process 800 of automating processes running in a user session via an RPA robot, IPC facilitator, and RPA driver running in a robot session, according to an embodiment of the invention. At 810, the process begins by starting a user session window. This may be, for example, a main window associated with an operating system running on the user computing system. Then at 820, the robot session window is launched as a child of the user session window. In some embodiments, for example, the robot session window may be initiated in response to the robot being initiated or otherwise initiated. Then at 830, a robot is initiated in a robot session.

The robot is executed at 840 and communicates with the IPC facilitator via IPC. The IPC facilitator can receive status notifications, acquired data, error messages, commands, or other communications from the robot via IPC. The IPC facilitator can also send messages to the robot (such as commands, status updates, error messages from the IPC facilitator or RPA driver, etc.).

At 850, using the data acquired from the robot, the RPA facilitator controls or otherwise causes the RPA driver to perform application interactions using the acquired data from the robot and/or other communications. For example, the RPA driver may be caused to move a mouse, click on a field, enter text, click on a button, navigate a menu, open or close, and apply or window, etc. At 860, in some embodiments, after the robot completes its execution, the robot session may end and the robot session window may be automatically closed.

Fig. 9 is a flow diagram illustrating a process 900 of automating processes running in a user session via RPA robots running in the robot session via direct variable modification, according to an embodiment of the invention. At 910, the process begins by launching a user session window. This may be, for example, a main window associated with an operating system running on the user computing system. Then at 920, a robot session window is launched as a child of the user session window. In some embodiments, for example, the robot session window may be initiated in response to the robot being initiated or otherwise initiated. The bot is then initiated in a bot session at 930, and at 940, the bot interacts with data associated with application(s) (e.g., Web pages, spreadsheets, ERP applications, sales applications, etc.) that are running in a user session from the bot session. For example, the bot may create, change, or delete stored information in computing system memory, such that the change occurs in commonly stored data that is accessed and used by application(s) in a user session. At 950, in some embodiments, after the robot completes its execution, the robot session may be ended and the robot session window may be automatically closed.

In some embodiments, multiple running automation processes (e.g., UI automation processes) may be isolated in a client session. This may allow these procedures to be invoked on demand from the master session via IPC, for example. In this manner, multiple RPA robot flows may be invoked on demand from a master session application, a master session RPA robot, another client session RPA robot, and so forth. An RPA robot running in a client session may also be able to collect data from the master session. In some embodiments, an RPA robot running in a client session may wait for some trigger from the host session before it performs its automation or part of the automation.

Fig. 10 is an architecture diagram 1000 illustrating an example of a multiple client session RPA bot interacting with a master session application in accordance with an embodiment of the present invention. In fig. 10, a master session 1010 and a client session 1020 are running. The client session 1020 includes a client session RPA robot 11030 and a client session RPA robot 21032.

Client session RPA robots 1030, 1032 with a separate IPC facilitator application 1040 or embedded RPA facilitator process or as a driver 10Subroutine 1052, a part of 50 communicates. In some embodiments, for example, drive 1050 may be drive 340 of FIG. 3. The IPC facilitator 1040 or 1052 causes the driver 1050 to interface with the master session application and/or associated application objects 1060 (e.g., for Microsoft Windows)

Component Object Model (COM) objects of an application). For example, driver 1050 may be caused to move a mouse, fill in form data, click one or more buttons, interact with menus, combinations thereof, and the like, within master session application 1060. In some embodiments, the RPA facilitator application 1040 can communicate directly with the master session application and/or associated application object 1060.

In some embodiments, the RPA robot running in the client session may be a manned automated robot. For example, an RPA robot running in a client session may require input from a user in a master session. This can be done via IPC with the master session robot, drive, etc.

Fig. 11 is a flow diagram 1100 illustrating a process for running a manned automated RPA robot in a client session in accordance with an embodiment of the present invention. In this embodiment, at 1120, the user utilizes a master session application 1110 that causes the client session RPA robot process to begin via IPC. In some embodiments, master session application 1110 may be an RPA assistant application that allows a user in a master session to interact with a manned RPA robot in a client session. For example, a dashboard of the RPA assistant application may show the current RPA processes supported by the robot, and the user may select the process to perform. In some embodiments, while the client session RPA bot is running, the RPA assistant application may display the steps and/or execution state of the client session workflow.

After the client session RPA robot starts at 1120, at 1130, the client session RPA robot runs the first part of its logic. After the first portion of the logic is executed at 1130, the client session RPA robot asks for user input at 1140. This may occur via messages in the application running in the master session that are presented via the IPC facilitator and driver interacting with the master session application, messages from the RPA bot running in the master session, and so on. The user then provides the requested input, which is sent to the client session RPA robot via IPC, and the second part of the logic is executed using the input at 1150. After the client session RPA robot flow ends at 1160, the master session application 1110 may receive a notification that the RPA robot flow has ended.

In some embodiments, portions of the automation may run in the master session, while other portions of the automation may run in the client session. For example, a master session application or RPA robot may perform certain operations in a master session and then cause the RPA robot running in a client session to perform another part of the automation when the master session application or robot reaches a certain point in its execution, receives a trigger, logic conditions are met, etc. The client session RPA robot may perform its automated portions in sequence or in parallel with the master session application or robot. The client session RPA robot may then provide the indication, requested data, execution results, etc. to the master session application or robot.

FIG. 12 is a flowchart 1200 illustrating the execution of a multi-robot collective workflow between a master session robot M1, a client session application A1, and a pair of client session robots C1 and C2 in accordance with an embodiment of the present invention. A1, C1, and/or C2 may be in the same client session or in different client sessions. M1 begins executing its workflow and reaches the activity of Call C1 to complete the workflow. In some embodiments, this may be performed by a "invoke workflow" activity having a flag set to perform an RPA procedure in the client session. M1 causes C1 to execute its workflow via IPC and invoke activity wait. During this time, M1 may perform other tasks in some embodiments.

After C1 completes its workflow, C1 notifies M1 via IPC, and M1 resumes execution until it reaches the activity that calls A1 to start execution or to complete a task, a process in a client session. M1 causes A1 to perform and/or execute tasks via IPC and call activity (e.g., "Call procedure" activity) to wait. In some embodiments, the calling procedure activity may have a flag set to perform a1 in the client session. In some embodiments, again, M1 may perform other tasks while waiting for a 1.

After A1 completes its process or task, A1 notifies M1 via IPC and M1 resumes execution until it reaches the activity that calls C2 to complete the workflow. M1 causes C2 to execute its workflow via IPC and invoke activity wait. In some embodiments, again, M1 may perform other tasks while waiting for C2.

After C2 completes its workflow, M1 is notified via IPC, and M1 resumes execution until it reaches the activity of calling C1 again to complete the workflow. This may be the same workflow activity that was previously performed, or may be a different workflow or set of activities. M1 causes C1 to execute its workflow via IPC and invoke activity wait. In some embodiments, again, M1 may perform other tasks while waiting for C1. After C1 completes its workflow, M1 is notified via IPC to resume execution until the M1 workflow is completed. In some embodiments, C1 may remain running after being initially invoked and still be running when invoked again by M1.

In accordance with the above, in some embodiments, M1 may include one or more "call workflow" activities and/or "call procedure" activities that may call RPA bot(s) (e.g., C1 and/or C2) and/or other procedures in the client session (e.g., a 1). Such activity may initiate the respective RPA robot(s) and/or process (es), communicate with the RPA robot(s) and/or process (es) already running, and/or the like. These activities may be created when the RPA developer develops a workflow for M1. In certain embodiments, the RPA robot(s) may invoke an RPA robot and/or process from a client session into a master session via an RPA facilitator or other suitable functionality.

During execution, M1 may communicate with a1, C1, and C2, and vice versa, via IPC. For example, M1 may send comments and requests to a1, C1, and C2, and M1 may receive status messages, results, error messages, and the like from a1, C1, and C2. In this manner, M1 may act as a master RPA robot that controls client session RPA robots and/or other processes (e.g., applications) in a client session. In certain embodiments, M1, a1, C1, and/or C2 may operate in parallel.

In some embodiments, the M1 may perform all interactions with the application running in the master session, which may eliminate the need for an IPC facilitator. Thus, C1 and C2 may perform various data acquisition and processing tasks, and M1 may perform UI interaction tasks. In certain embodiments, at least some of the workflow activities of M1, C1, and/or C2 may be performed in parallel. One such example is shown in flowchart 1300 of FIG. 13, where C1 is executed based on logic of M1, and execution occurs in parallel. Communications may be sent between M1 and C1 during the section (section) of C1. However, in some embodiments, C1 may complete its execution independently, and may complete its workflow execution after M1 ends.

In some embodiments, all RPA robot activities are running in a client session. The RPA robot may be invoked by a robot effector from a master session, such as, for example

A robotic tray, an IPC facilitator, another application, etc. The results of the RPA robot's execution and other communications may be provided to the robot actuators or other applications via IPC.

In some embodiments, a standardized communication format may be used for IPC communications between a client session and a master session. For example, the master session and client session processes may exchange XAML data. This helps to ensure that the communication is in a format that the corresponding process can recognize.

In some embodiments, IPC may be used between RPA processes running in the same session for synchronization purposes. For example, multiple RPA processes may run in the background in parallel with the foreground process in the master session. IPC may provide a synchronization mechanism to exchange information between a background process and a foreground or main process.

In some embodiments, a driver (e.g., driver 340 of fig. 3) will be loaded during the main RPA workflow. Communication between the child RPA process and the main RPA workflow process may occur via IPC. This allows the master RPA workflow process to perform operations for the client session robot based on IPC information with the driver.

In some embodiments, the client session RPA bot may work with an application that is opened in the master session. For example, a client session RPA robot may open with a session master

The spreadsheets work together. This may occur, for example, between the client session RPA robot and the RPA facilitator via IPC. The RPA facilitator can implement the driver based on information provided by the client session RPA robot (e.g., enter data into a spreadsheet, modify a table value, etc.)

Interaction(s) of the spreadsheet. In some embodiments, the activity may be included in an RPA workflow that automatically captures application data between two sessions (e.g., such as

Activity, Web browser activity, etc.). In this manner, the client session RPA robots can essentially work with applications running in the master session as if they were running in the RPA robot's own session. Indeed, in some embodiments, an RPA developer may not know that the workflow or a portion thereof that it is developing is to be executed in a client session.

In some embodiments, for example, when designing an RPA bot using a designer application, an RPA developer may set a client session flag on a "invoke workflow" activity or a "invoke procedure" activity. The invoke workflow activity invokes the RPA workflow. Invoking the procedure activity executes a procedure that is available to the local machine. Setting this flag may cause an RPA workflow or process to be invoked in a client session rather than a master session.

In some embodiments, the execution of the client session RPA robot may be based on a trigger from the master session application and/or the RPA robot. Fig. 14 is a flow diagram 1400 illustrating execution of a client session RPA bot based on a trigger for a master session application in accordance with an embodiment of the present invention. The master session application a1 (which may be an RPA robot in some embodiments) starts executing and waits for a trigger. After receiving the trigger, A1 begins executing the client session RPA robot C1 via IPC. Communications may be sent between a1 and C1 during the segment of C1. Then, a1 waits for another trigger and C1 executes its workflow.

FIG. 15 is a flowchart 1500 illustrating a process for performing inter-session automation in accordance with an embodiment of the present invention. At 1510, the process begins by executing an RPA robot in a master session of the computing system. At 1520, the RPA robot or application is invoked or launched in the client session by the master session RPA robot via IPC. At 1530, messages are communicated between the client session RPA robot or application and the master session RPA robot via IPC. In some embodiments, the master session RPA robot suspends its execution until the end of the execution of the client session RPA robot or application at 1540.

At 1550, the result(s) of the execution of the client session RPA robot or application is received by the master session RPA robot via IPC. The master session RPA robot then uses the result(s) at 1560 to complete at least a portion of the workflow of the master session RPA robot, interact with an application or application object running in the master session, or both. In some embodiments, at 1570, the process repeats for the next client session RPA robot or client session application, and the process returns to step 1520.

In some embodiments, the client session RPA robot, the client session application, or both are invoked or initiated two or more times in the workflow of the master session RPA robot. In some embodiments, the client session application is invoked or initiated via an invoking process activity of the workflow of the master session RPA robot, the client session RPA robot is invoked or initiated via an invoking workflow activity of the workflow of the master session RPA robot, or both. In some embodiments, invoking the procedural activity comprises indicating that the client session application is running in the client session, invoking the workflow activity comprises indicating that the client session RPA robot is running in the client session, or both. In some embodiments, the client session application, the client session RPA robot, or both, continue to run after providing the results. In some embodiments, IPC communications between the master session RPA robot and the client session RPA robot and/or the client session application are in a standardized communication format.

The process steps performed in fig. 8-15 may be performed by a computer program encoding instructions for execution by a processor(s) to perform at least a portion of the process (es) described in fig. 8-15 according to an embodiment of the invention. The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to: a hard disk drive, a flash memory device, RAM, magnetic tape, and/or any other such medium or combination of media for storing data. The computer program may include coded instructions for controlling processor(s) of a computing system, such as processor(s) 510 of computing system 500 of fig. 5, to implement all or a portion of the process steps described in fig. 8-15, and may also be stored on a computer-readable medium.

The computer program may be implemented in hardware, software or a hybrid implementation. Computer programs may be comprised of modules that are in operable communication with each other and are designed to communicate information or instructions for display. The computer program may be configured to operate on a general purpose computer, ASIC or any other suitable device.

It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention.

The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, reference throughout this specification to "certain embodiments," "some embodiments," or similar language means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language throughout this specification do not necessarily all refer to the same group of embodiments and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

It should be noted that reference throughout this specification to features, advantages, or similar language does not imply that all of the features and advantages that may be realized with the present invention should be or are in any single embodiment of the invention. Rather, language referring to the features and advantages is understood to mean that a specific feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussion of the features and advantages, and similar language, throughout this specification may, but do not necessarily, refer to the same embodiment.

Furthermore, the described features, structures, or characteristics of the invention may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize that the invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.

One of ordinary skill in the art will readily appreciate that the invention as described above may be practiced with steps in a different order and/or with hardware elements in configurations other than those disclosed. Thus, while the invention has been described based upon these preferred embodiments, it would be apparent to those of ordinary skill in the art that certain modifications, variations, and alternative constructions would be apparent, while remaining within the spirit and scope of the invention. Therefore, to ascertain the metes and bounds of the invention, the appended claims should be referenced.

Claims (30)

1. A computer-implemented method, comprising:

executing a Robot Process Automation (RPA) robot in a master session of a computing system;

invoking or starting, by the master session RPA robot, an RPA robot in a client session via inter-process communication (IPC);

receiving, by the master session RPA robot, one or more messages from the client session RPA robot via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the client session RPA robot to: completing at least a portion of a workflow of the master session RPA robot, interacting with an application or application object running in the master session, or both.

2. The computer-implemented method of claim 1, further comprising:

invoking or launching an application in the client session by the master session RPA robot via IPC;

receiving, by the master session RPA robot, one or more messages from the client session application via IPC during and/or after execution of the task of the application; and

using, by the master session RPA robot, results from the one or more messages received from the client session application to: completing at least a portion of the workflow of the master session RPA robot, interacting with the application or application object running in the master session, or both.

3. The computer-implemented method of claim 2, further comprising:

suspending execution of the master session RPA robot until the client session application completes execution.

4. The computer-implemented method of claim 2, wherein the client session application is invoked or launched two or more times in the workflow of the master session RPA robot.

5. The computer-implemented method of claim 2, further comprising:

invoking or starting, by the master session RPA robot, an additional application in the client session via IPC;

receiving one or more messages from the additional client session application via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the additional client session application to: completing a portion of the workflow of the master session RPA robot, interacting with the application or application object running in the master session, or both.

6. The computer-implemented method of claim 2, wherein the client session application is invoked or launched via a calling process activity of the workflow of the master session RPA robot.

7. The computer-implemented method of claim 6, wherein the calling process activity comprises a flag indicating that the client session application is running in the client session.

8. The computer-implemented method of claim 2, wherein the client session application continues to run after the task is completed.

9. The computer-implemented method of claim 2, wherein IPC communications between the master session RPA robot and the client session application are in a standardized communication format.

10. The computer-implemented method of claim 1, further comprising:

suspending execution of the master session RPA robot until the client session RPA robot completes execution.

11. The computer-implemented method of claim 1, further comprising:

calling or starting an additional RPA robot in the client session by the main session RPA robot through IPC;

receiving one or more messages from the additional client session RPA robot via IPC; and

using, by the master session RPA robot, results from one or more messages received from the additional client session RPA robot to: completing a portion of the workflow of the master session RPA robot, interacting with the application or application object running in the master session, or both.

12. The computer-implemented method of claim 1, wherein the client session RPA robot is invoked or launched two or more times in the workflow of the master session RPA robot.

13. The computer-implemented method of claim 1, wherein the client session RPA robot is invoked or initiated via an invoking workflow activity of the workflow of the master session RPA robot.

14. The computer-implemented method of claim 13, wherein the invoking workflow activity comprises a flag indicating that the client session RPA robot is running in the client session.

15. The computer-implemented method of claim 1, wherein the client session RPA robot continues to run after the results are provided to the master session RPA robot.

16. The computer-implemented method of claim 1 wherein the IPC communication between the master session RPA robot and the client session RPA robot is in a standardized communication format.

17. A computer-implemented method, comprising:

executing a Robot Process Automation (RPA) robot in a master session of a computing system;

invoking or launching an application in a client session by a master session RPA robot via inter-process communication (IPC);

receiving, by the master session RPA robot, one or more messages from a client session application via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the client session application to: completing at least a portion of a workflow of the master session RPA robot, interacting with an application or application object running in the master session, or both.

18. The computer-implemented method of claim 17, further comprising:

suspending execution of the master session RPA robot until the client session application completes execution.

19. The computer-implemented method of claim 17, wherein the client session application is invoked or launched two or more times in the workflow of the master session RPA robot.

20. The computer-implemented method of claim 17, further comprising:

invoking or starting, by the master session RPA robot, an additional application in the client session via IPC;

receiving one or more messages from the additional client session application via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the additional client session application to: completing a portion of the workflow of the master session RPA robot, interacting with the application or application object running in the master session, or both.

21. The computer-implemented method of claim 17, wherein the client session application is invoked or launched via a calling process activity of the workflow of the master session RPA robot.

22. The computer-implemented method of claim 21, wherein the invoking procedure activity comprises a flag indicating that the client session application is running in the client session.

23. The computer-implemented method of claim 17, wherein the client session application continues to run after the task is completed.

24. A computer-implemented method, comprising:

executing a Robot Process Automation (RPA) robot in a master session of a computing system;

invoking or starting, by a master session RPA robot, the RPA robot in a client session via inter-process communication (IPC);

invoking or launching an application in the client session by the master session RPA robot via IPC;

receiving, by the master session RPA robot, one or more messages from the client session RPA robot and the client session application via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the client session RPA robot and the client session application to: completing at least a portion of a workflow of the master session RPA robot, interacting with one or more applications and/or application objects running in the master session, or both.

25. The computer-implemented method of claim 24, further comprising:

suspending execution of the master session RPA robot until the client session RPA robot, the client session application, or both, complete execution.

26. The computer-implemented method of claim 24, wherein the client session application, or both are invoked or launched two or more times in the workflow of the master session RPA robot.

27. The computer-implemented method of claim 24, further comprising:

invoking or starting, by the master session RPA robot, an additional application, an additional RPA robot, or both, in the client session via IPC;

receiving one or more messages from the additional client session application, the additional RPA bot, or both via IPC; and

using, by the master session RPA robot, results from the one or more messages received from the additional client session application, the additional client session RPA robot, or both, to: completing a portion of the workflow of the master session RPA robot, interacting with at least one of the one or more applications and/or application objects running in the master session, or both.

28. The computer-implemented method of claim 24, wherein the client session application is invoked or launched via an invoking process activity of the workflow of the master session RPA robot, or the client session RPA robot is invoked or launched via an invoking workflow activity of the workflow of the master session RPA robot, or both.

29. The computer-implemented method of claim 28, wherein the invoking procedure activity comprises a flag indicating that the client session application is running in the client session, or the invoking workflow activity comprises a flag indicating that the client session RPA robot is running in the client session, or both.

30. The computer-implemented method of claim 24, wherein the client session application, the client session RPA robot, or both, continue to run after providing the result.

Images