JP6452882B1 - System, method and program for automating business processes involving operation of web browser - Google Patents

Abstract

A system for automating a business process involving operation of a web browser is provided. A user accesses a website provided by a first server. At this time, the user terminal receives a creation program for creating RPA data in the first terminal from the first server. The user inputs the URL of the target web page in the input field and clicks the “display” button. When all necessary operations have been specified, an operation history is displayed in the operation column of the website. When the user clicks the “Save” button in the operation column, RPA data corresponding to the operation history is generated. [Selection] Figure 2

Description

  The present invention relates to a system, method, and program for automating business processes, and more particularly, to a system, method, and program for automating business processes that involve operation of a web browser.

  The use of robots, which have been introduced in factories and the like, is now spreading to office work. This is an operation that automates a business process by storing operations performed by a human on a computer, and is called robotic process automation (RPA). Businesses that can routinely operate, for example, can be automated even if it spans multiple application software, and can be repeated faster and faster than humans and without making mistakes. Have been bathed.

  In the future, as various software used by companies shift from on-premise to cloud-type, operations on RPA will be a large percentage of operations on web browsers.

  However, conventional RPA software is not mainly used for operations on web browsers, and there is not much room for improvement.

  The present invention has been made in view of such problems, and an object thereof is to provide a system, a method, and a program for automating a business process that involves operation of a web browser.

  In order to achieve such an object, a first aspect of the present invention is a method for automating a business process involving an operation of a web browser, and an operation designated by a human being is read as data that can be read by a computer. To describe, in a web browser, displaying a captured image of a web page to be operated, receiving a selection for a selectable element on the captured image, and selecting the selected web browser Displaying one or more selectable operations associated with the element; receiving as a designation a selection for any of the one or more operations; and for the selected one or more elements Generating one or more data describing the specified operation. And features.

  According to a second aspect of the present invention, in the first aspect, each element in the web page corresponds to a position on the captured image corresponding to a position on the display of the element on the web page. It is possible to select an object by arranging it.

  According to a third aspect of the present invention, in the second aspect, the object has a rectangular shape surrounding a corresponding element.

  According to a fourth aspect of the present invention, in the third aspect, the object is transparent before selection and a boundary line is displayed after selection.

  According to a fifth aspect of the present invention, in any one of the first to fourth aspects, when the designated operation causes a change in display on the web page, a captured image of the web page to be operated Is displayed again.

  According to a sixth aspect of the present invention, in the fifth aspect, the designated operation is a click or tap of a link or pull-down menu.

  According to a seventh aspect of the present invention, in the fifth aspect, the designated operation is an input to an input form.

  According to an eighth aspect of the present invention, in any one of the first to fourth aspects, the designated operation is output to a file or input from a file.

  According to a ninth aspect of the present invention, in any one of the first to eighth aspects, an operation can be specified for a plurality of elements on the same or different web pages.

  According to a tenth aspect of the present invention, there is provided a program for causing a computer to execute a method for automating a business process involving operation of a web browser, wherein the method performs an operation designated by a human by a computer. Displaying in a web browser a captured image of a web page to be manipulated, receiving a selection for a selectable element on the captured image, to the web browser to describe as readable data; Displaying one or more selectable operations associated with the selected element, and receiving as a designation a selection for any of the one or more operations. Or one or more that describe the specified operations for multiple elements And transmitting the data.

  According to an eleventh aspect of the present invention, there is provided a method for automating a business process involving an operation of a web browser, in order to describe a human-designated operation as data readable by a computer, Receiving a captured image and source code of a web page to be operated; analyzing the source code to determine the position of each element in the source code; and displaying each element on the web page Identifying a position; and transmitting the captured image and the position of each element in the source code and the position on the display to a user terminal. The captured image received by the user terminal in a web browser. Displaying the captured image Receiving a selection for the selectable element, displaying in the web browser one or more selectable operations associated with the selected element, and any of the one or more operations Receiving the selection as a designation, and generating one or more data describing the designated operation for the selected element or elements and sending it to the server, the web page Each of the elements can be selected by arranging a corresponding object at a position on the captured image corresponding to a position on the display of the element on the web page.

  According to a twelfth aspect of the present invention, in the eleventh aspect, the position in the source code is indicated by an absolute path.

  According to a thirteenth aspect of the present invention, in the eleventh or twelfth aspect, the position on the display is a position when a horizontal width of the web page is determined as a predetermined width.

  According to a fourteenth aspect of the present invention, in any one of the eleventh to thirteenth aspects, the position on the display is indicated by coordinates based on a predetermined point in the web page. To do.

  According to a fifteenth aspect of the present invention, in the fourteenth aspect, the predetermined point is a left end or a right end of an upper end of the web page.

  According to a sixteenth aspect of the present invention, in any one of the eleventh to fifteenth aspects, the one or more data is in a JSON format.

  According to a seventeenth aspect of the present invention, in the sixteenth aspect, the server converts the one or more data into a block-style YAML format and stores the data.

  According to an eighteenth aspect of the present invention, in any one of the eleventh to seventeenth aspects, the server is one or a plurality of instances on a cloud.

  According to a nineteenth aspect of the present invention, there is provided a program for causing a server and a user terminal to execute a method for automating a business process accompanied by an operation of a web browser, the method including an operation designated by a human The server receives a captured image and source code of a web page to be operated, and analyzes the source code to position each element in the source code Identifying the display position of each element on the web page, and transmitting the captured image and the position of each element in the source code and the display position to a user terminal. Including the user terminal in a web browser, Displaying the captured image received, receiving a selection for a selectable element on the capture image, and causing the web browser to select one or more operations associated with the selected element. Generating one or more data describing the step of displaying, the step of receiving a selection for one of the one or more operations as a designation, and the operation designated for the selected one or more elements Each element in the web page arranges a corresponding object at a position on the captured image corresponding to a position on the display of the element on the web page. It is possible to select with.

  According to a twentieth aspect of the present invention, there is provided a system for automating a business process accompanied by an operation of a web browser, wherein an operation designated by a human being is described as data readable by a computer. The web page capture image and source code are received, the source code is analyzed, the position of each element in the source code and the position of each element on the web page are specified, the capture image and each A server that transmits a position of the element in the source code and a position on the display to a user terminal; and a web browser that displays the received captured image, receives a selection for an element selectable on the captured image, and In the web browser, the selected element Display one or more selectable operations associated with the selected object, receive a selection for one of the one or more operations as a specification, and specify the operation for the selected one or more elements A user terminal that generates one or a plurality of data that describes and transmits the data to the server, and each element in the web page corresponds to a display position of each element on the web page in the user terminal The object can be selected by placing a corresponding object at a position on the captured image.

  According to one aspect of the present invention, a user can specify an operation for each element to be automated via a captured image of a web page to be operated, and web pages that cannot acquire a captured image are limited. Therefore, it is possible to easily target a wide range of web pages as RPA.

It is a figure which shows the system for automation concerning the 1st Embodiment of this invention. It is a figure which shows the flow of a user's work procedure for the robot preparation concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention. It is a figure which shows the flow of the robot preparation procedure concerning the 1st Embodiment of this invention.

  There are two steps to create a so-called “robot” and execute it in order to cause a computer to automatically execute a business process involving the operation of a web browser. Here, the term “robot” is ambiguous, such as a program or code executable by a computer in which an operation performed by a human is described, or data describing an operation performed by a human that can read a program executable by the computer. In the present specification, it is used only for the purpose of schematic explanation.

  Further, if an operation performed by a human or a specified operation can be described as data that can be interpreted or read by a computer (hereinafter also referred to as “RPA data”), based on the data, if necessary, for example, described later Such as Javascript (registered trademark) in a format that can be executed by an application for operating a web browser such as Selenium, or converted into a format that can be interpreted or read, or that can be executed or interpreted or read by the application. A detailed program can be generated and the designated operation can be automatically executed, so that details thereof will not be described.

  The present invention allows a person who does not have a high level of programming ability to interpret an operation performed by a human or a specified operation by a computer in order to enable automation of operations on various websites. It is easy to describe as simple data. In other words, the present invention can facilitate the creation of a so-called “robot”.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 shows a configuration for automation according to the present embodiment. The server system 100 includes a first server 110 that communicates with the user terminal 130, and a second server 120 that accesses a target website based on a command from the first server 110. The website is provided by a third server 140 on the Internet. Each of the first server 110 and the second server 120 may be an instance on the cloud, and as described later, an instance corresponding to the second server 120 may be generated for each of one or more robots. In the following, although this configuration will be described as an example, the second server 120 may be the same server without being a separate server from the first server 110. In this case, between these servers, No data is exchanged. The server system 100 and the user terminal 130 may be collectively referred to as “system”.

  The first server 110 includes a communication unit 111 such as a communication interface, a processing unit 112 such as a processor and a CPU, and a storage unit 113 including a storage device or a storage medium such as a memory and a hard disk. It can be configured by executing a program for performing each process described later. The first server 110 may include one or a plurality of servers or instances, and the program may include one or a plurality of programs. The first server 110 may be recorded on a computer-readable storage medium and stored in a non-unique manner. Can be a transient program product.

  Although devices, servers or instances other than the first server 110 are not individually illustrated, a communication unit, a processing unit, and a storage unit are provided in the same manner as the first server 110, and one or more programs are processed by the processing unit. The required function is realized by executing the above.

Creation of RPA Data A procedure for creating RPA data in this embodiment will be described. First, the user accesses the website 300 provided by the first server 110 from the user terminal 130 (S201). At the time of access, login is performed as necessary. At this time, the user terminal 130 receives from the first server 110 or the system 100 a creation program for creating RPA data using the web browser in the first terminal 130. The creation program can be, for example, JavaScript, and the user terminal 130 stores operations performed by the user thereafter, and updates the display of the website 300 as shown in FIG. 4 and subsequent drawings according to the user operation. .

  Next, the user gives a title to the newly created robot on the website 300 and clicks or taps the “Create” button (S202). In the example of FIG. 3, the title “test robot” is given, and the Excel (registered trademark) file “test” is used as an output file for outputting data acquired as a result of an operation on the target website. "Robot output file.xlsx" is associated with the robot.

  The first server 110 that has received the request for creating a new robot from the user terminal 130 generates the second server 120 as an instance for the robot. An instance may be generated for each robot, or a generated instance may be assigned, or a predetermined number of robots such as four or five may be assigned to each instance. In this case, if there is no assignable instance, a new instance is generated. Further, the first server 110 stores the associated output file in association with the robot in a storage device or storage medium accessible from the storage unit 113 or the first server 110 via the network.

  Next, the user inputs the URL of the target web page in the URL input field 401 and clicks or taps the “display” button 402 (S203). In the example of FIG. 4, an image of a news page of a website of a certain company is displayed in the display field 310 of the website 300. As will be described later, each element of the news page can be selected on the image based on the analysis result of the source code. In the website 300, in addition to the display column 310, an operation column 320 is displayed. In the operation column 320, a history of operations performed by the user is left. Each operation is composed of an operation type and an object, and here, the URL that is the object is shown together with the type “open URL”.

  The first server 110 that has received the web page display request from the user terminal 130 causes the second server 120 to start a browser and display the URL. The second server 120 can operate the browser via an application such as Selenium. Here, it is not always necessary to start the browser in the second server 120 at this time, and the second server 120 displays the URL on the browser, but does not display it on the display. Not necessary.

  If the URL is successfully displayed, the second server 120 captures the displayed web page as an image, and delivers it to the first server 110 together with its source code. The first server 110 transmits the received image and the analysis result of the source code to the user terminal 130, and the user terminal 130 updates the display field 310 of the website 300 as shown in FIG. The update is performed by the above-described generation program based on the image and the analysis result. The analysis is performed by the second server 120, and the analysis result can be passed to the first server 110. Details of the analysis will be described later.

  When the user selects any element 501 in the target web page displayed in the display column 310 (S204), a list of possible operations for the element is displayed in the operation column 320. In the example of FIG. 5, “acquire text”, “extract repetition”, “click”, “move the mouse cursor”, and “save link destination” are shown as options. When “extract repetition” is selected here, the type of operation is determined (S205-1).

  When “extract repetition” is the type of operation, an element having the same or similar path as the selected element 501 is selected as the operation target based on the analysis result of the source code. In FIG. 6, elements 502 to 506 having the same class div tag are automatically selected. Here, the name of a repetitive operation is determined by giving a name and clicking or tapping the “OK” button (S205-2).

  In FIG. 6, an element having the same or similar path as the selected element 501 is automatically selected, but is shown by, for example, highlighting a repeated element candidate for the selected element 501. Depending on the fact that the user has selected the second element, it is also possible to select an element having an absolute path common to both elements as a repetition target.

  When “extract repetition” is the type of operation, the target of the repetition operation has been determined by the operations so far, but the content of the operation to be repeated is undecided. Therefore, the user clicks or taps “+” in the operation column 320 to display again a list of operations that can be selected for the element in the operation column 320 and selects “acquire text” ( S205-3). Then, as shown in FIG. 7, by tapping or clicking the “confirm” button in the operation column 320, the entire repetitive operation is determined. The details for confirming the repetitive operation can naturally be expressed as various UIs in addition to the “confirm” button.

  In addition to “Extract Repeat”, for example, when an operation of “click” is selected, the screen transitions if the link is clicked, and the option is displayed if the click is a pull-down menu. Therefore, a new URL is reopened and the display field 310 of the website 300 is updated. Further, without changing the screen, another element may be selected and the operation type and target may be designated in the same manner.

  When the operation on the website is completed, if the user is acquiring text as in this example, the user selects the operation “output to file” shown in FIG. An output format is designated (S206). The output file association may be performed here. As shown in FIG. 9, when the output file is a spreadsheet, the output start cell of the extracted text can be specified, but the output format is not necessarily specified. In addition, an output file provided in advance may be used as a template, and a storage location, a file name, and the like may be separately specified.

  When all necessary operations have been specified, an operation history is displayed in the operation column 320 of the website 300. The user completes the creation of the robot by tapping or clicking the “Save” button in the operation column 320 (S207). At this time, the user terminal 130 generates RPA data corresponding to an operation history of an operation on the selected element and transmits it to the first server 110, and the first server 110 converts the data into the data as necessary. To remember.

  For example, the RPA data transmitted from the user terminal 130 can be in the JSON format, and the converted data in the first server 110 or the system 100 can be in the block style YAML format. Further, transmission from the user terminal 130 may be performed every time an operation is performed, instead of waiting for completion of designation of all operations.

Taking the above-described operation of “opening URL” as an example, data of the following format is stored in the user terminal 130 at the time of the operation, for example.
{
type: 'open_url',
id: 'd0e0ff67-980f-4321-8564-458087f12515',
browser_size:
width: 1280,
height: 1024,
url: 'https://www.xxxx.co.jp/news.html'
comment:
}

When other operations are performed, data in the same format is generated and stored in the following order in the order in which the operations were performed.When all operations are completed or when each operation is completed, The parallel data stored in the user terminal 130 can be transmitted to the first server 110.
[
{
type: 'open_url',
... (omitted)
},
{
type: 'foreach',
... (omitted)
}
]

The first server 110 can convert the received array data into the following YAML format data and store it as a character string.
settings:
default_error_handling: fail_and_stop
code:
-type: open_url
id: d0e0ff67-980f-4321-8564-458087f12515
browser_size:
width: 1280
height: 1024
url: https://www.xxxx.co.jp/news.html
comment:
-type: foreach
id: b0a04b57-2648-491f-8479-32fa3866690d
(Omitted)

  In addition, in the present specification, additional information is provided unless there is a description of “only”, such as “based only on XX”, “according to XX only”, and “only XX”. Note that it is also assumed that

  Also, just in case, even if there is an aspect of performing an operation different from the operation described in this specification in any method, program, terminal, device, server, or system (hereinafter “method etc.”), each of the present invention The aspect is directed to the same operation as any of the operations described in this specification, and there is an operation different from the operation described in this specification. It is added that it is not outside the scope of the embodiment.

Details of Source Code Analysis The source code analysis of the target web page performed in the first server 110 includes specifying the position of each element in the source code and specifying the position of each element on the web page. . Moreover, at least one of the tag name and attribute of each element and the height and width on display can be included.

  In the user terminal 130, the generation program, for each element in the target web page based on the position information, at a position on the captured image corresponding to the position on the display of each element on the web page, Place the corresponding object.

  For example, as shown in FIG. 5, the object can be a rectangular frame surrounding the corresponding element. The object can be transparent before selection and a boundary line can be displayed after selection. .

  The position of each element in the source code can be indicated by an absolute path, and the display position of each element on the web page is the position when the width of the web page is set to a predetermined width, for example, 1280 pixels. can do. The position can be represented by coordinates with the origin at the upper left or upper right corner of the web page, that is, the upper left or right edge when the web page is scrolled to the top. The coordinates can be the coordinates of any vertex such as the upper left or upper right of each element.

(Second Embodiment)
In the first embodiment, the same operation is repeated for elements that repeatedly appear, but different operations designated for the same element can be repeated. For example, it is possible to repeat an operation of opening a search engine, inputting a keyword input in a spreadsheet in advance into an input form, and acquiring text from the search result.

100 system 110 first server 111 communication unit 112 processing unit 113 storage unit 120 second server 130 user terminal 140 third server

Claims (10)

A method for automating business processes that involve web browser operations to describe human-designated operations as computer-readable data.
Displaying a captured image of the web page to be manipulated in a web browser;
Receiving a selection for a selectable element on the captured image;
Displaying in the web browser one or more selectable operations associated with the selected element;
Receiving as a designation a selection for any of the one or more operations;
Generating one or more data describing a specified operation for the selected one or more elements;
Each element in the web page is selectable by placing a corresponding object at a position on the captured image corresponding to a display position of each element on the web page. .   The method of claim 1, wherein the object is a rectangular shape surrounding a corresponding element.   3. The method according to claim 1, wherein when the designated operation causes a display change in the web page, a captured image of the web page to be operated is displayed again. 4.   4. The method according to claim 1, wherein the designated operation is output to a file or input from a file. The method according to any one of claims 1 to 4, characterized in that the operation for a plurality of elements of the same or different web pages can be specified. A program for causing a computer to execute a method for automating a business process involving an operation of a web browser, wherein the method is for describing a human-designated operation as data readable by a computer.
Displaying a captured image of the web page to be manipulated in a web browser;
Receiving a selection for a selectable element on the captured image;
Displaying in the web browser one or more selectable operations associated with the selected element;
Receiving as a designation a selection for any of the one or more operations;
Generating one or more data describing a specified operation for the selected one or more elements;
Each element in the web page can be selected by arranging a corresponding object at a position on the capture image corresponding to a position on the display of the element on the web page. . A method for automating business processes that involve web browser operations to describe human-designated operations as computer-readable data.
The server
Receiving a captured image and source code of a web page to be operated;
Analyzing the source code to determine the position of each element in the source code;
Identifying the display position of each element on the web page;
Transmitting the captured image and the position of each element in the source code and the position on the display to a user terminal,
The user terminal is
Displaying the received captured image on a web browser;
Receiving a selection for a selectable element on the captured image;
Displaying in the web browser one or more selectable operations associated with the selected element;
Receiving as a designation a selection for any of the one or more operations;
Generating one or more data describing the specified operation for the selected one or more elements and transmitting to the server;
Each element in the web page is selectable by placing a corresponding object at a position on the captured image corresponding to a display position of each element on the web page. .   The method of claim 7, wherein the server is one or more instances on a cloud. A program for causing a server and a user terminal to execute a method for automating a business process involving an operation of a web browser, wherein the method describes a human-designated operation as data readable by a computer In addition,
The server
Receiving a captured image and source code of a web page to be operated;
Analyzing the source code to determine the position of each element in the source code;
Identifying the display position of each element on the web page;
Transmitting the captured image and the position of each element in the source code and the position on the display to a user terminal,
The user terminal is
Displaying the received captured image on a web browser;
Receiving a selection for a selectable element on the captured image;
Displaying in the web browser one or more selectable operations associated with the selected element;
Receiving as a designation a selection for any of the one or more operations;
Generating one or more data describing the specified operation for the selected one or more elements and transmitting to the server;
Each element in the web page can be selected by arranging a corresponding object at a position on the capture image corresponding to a position on the display of the element on the web page. . A system for automating business processes that involve web browser operations, in order to describe human-designated operations as computer-readable data.
The captured image and source code of the web page to be operated are received, the source code is analyzed, the position of each element in the source code and the position of each element on the web page are identified, and the capture A server for transmitting an image and a position of each element in the source code and a position on the display to a user terminal;
Display the received captured image in a web browser, receive a selection for a selectable element on the captured image, and select one or more selectable operations associated with the selected element in the web browser Display, receive as selection a selection for one of the one or more operations, generate one or more data describing the specified operation for the selected one or more elements, and generate the server With a user terminal that sends to
Each element in the web page can be selected by arranging a corresponding object at a position on the capture image corresponding to a position on the display of the element on the web page in the user terminal. A system characterized by

Images