JP2020009279A - Information processing program, information processing device and information processing method - Google Patents

Abstract

To provide an information processing program capable of performing error processing in the case that an operation different from an operation desired by a user is probably executed or has been executed.SOLUTION: An operation storage module 115 of an information processing device stores a series of operations, a verification area storage module 130 stores area information showing an unchanging area and information in the area with a screen before or after performing an operation as an object, an operation execution module 120 executes an operation stored in an operation storage module, and an error processing module 140 performs error processing by using the area information from the screen before or after execution by execution means in the case that information in the area having no possibility of changing does not coincide with information stored in the verification area storage module.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing program, an information processing apparatus, and an information processing method.

  Patent Literature 1 has an object to provide an automatic operation agent capable of responding to a change in an operation target AP or an application target PC, and to provide a user by an image recognition unit that performs an image region identification process using an image region division engine. The processing result of the action unit of the operation is recorded in the action unit of the user operation by the script synchronization unit together with the processing result of the action unit of the user operation by the control control unit that performs the object specifying process, and the image recognition unit and control control are performed according to the record By operating the unit to replay the user's operation, the target of operation is matched by the action unit of the user operation by the specific result inquiry unit and cross-checked to detect the effect of the change of the target AP and the target PC. It has been disclosed.

  Japanese Patent Application Laid-Open No. H11-163873 aims to make it easy to specify the cause of the occurrence of a GUI screen transition error or a generation error when the GUI screen is generated by a GUI application. , A GUI screen design specification to determine a GUI screen generation error, correlate with a test script, GUI application execution location information, GUI application trace information generated when an error occurs, an error GUI screen, etc. It is disclosed that the GUI screen test result information including the above is generated, whereby the error of the GUI screen and the location of the source code being executed at the time of occurrence of the error are clarified, and the cause of the error can be easily identified. ing.

JP-A-2005-005245 JP 2010-152429 A

There is an automation tool called RPA (Robotic Process Automation) for automating operations. Specifically, by registering a plurality of operations by the user (for example, operations using a mouse and a keyboard on a screen, operations using a finger on a touch panel, operations using a pen, and the like), and executing the registered operations, The second and subsequent fixed operations can be easily executed.
However, when the layout of the operation target is changed on the screen, the position on the screen where the user wants to operate is different from the registered position on the screen. Therefore, an operation different from the operation desired by the user is performed.
An object of the present invention is to provide an information processing program, an information processing apparatus, and an information processing method that can perform error processing when an operation different from an operation desired by a user is likely to be performed or is performed. I have.

The gist of the present invention to achieve this object lies in the inventions in the following items.
According to the first aspect of the present invention, a computer stores a first storage unit that stores a series of operations, area information indicating an unchanged area, and information in the area for a screen before or after the operation. The second storage means, the execution means for executing the operation stored in the first storage means, and the area information from the screen before or after execution by the execution means, which should not change using the area information If the information in the area does not match the information stored in the second storage means, the information processing program is to be caused to function as an error processing means for performing error processing.

  The invention according to claim 2 is the information processing program according to claim 1, wherein an image in the area is used as the information in the area.

  The invention according to claim 3 is the information processing program according to claim 1, wherein the information in the area uses characteristic information in the area.

  According to a fourth aspect of the present invention, the area information indicating an unchanged area and the information in the area are extracted from the screen before executing the operation at a different time from the screen before executing the operation. The information processing program according to claim 1, further comprising a control unit configured to perform control to cause the storage unit to store the information.

  The invention according to claim 5, wherein region information indicating a region that has not changed and information in the region are extracted from the screen after performing the operation at a different time from the screen after performing the operation. The information processing program according to claim 1, further comprising a control unit configured to perform control to cause the storage unit to store the information.

  According to a sixth aspect of the present invention, region information indicating an unchanged region and information in the region are extracted from a screen before performing the operation and a screen after performing the operation, and the extracted information is stored in the second storage unit. The information processing program according to claim 1, further comprising a control unit configured to perform control to store the program.

  The invention according to claim 7, wherein the second storage means stores an internal state after the operation is performed, and the error processing means stores the internal state after the operation and the second storage means. The information processing program according to claim 1, wherein when the internal states stored in the information do not match, error processing is performed.

  The information processing apparatus according to claim 1, wherein the error processing unit performs a process of detecting an error before at least a first operation among operations performed by the execution unit is performed. It is a program.

  The invention according to claim 9 is the information processing program according to claim 8, wherein the error processing means performs a process of detecting an error before each operation executed by the execution means is executed.

  According to a tenth aspect of the present invention, the error processing unit detects an error after at least the last operation of the operations performed by the execution unit has been performed and before data storage processing by the operation is performed. The information processing program according to claim 1, wherein the information processing program performs a detection process.

  The invention according to claim 11, wherein the error processing unit performs a process of detecting an error after each operation executed by the execution unit has been performed and before data storage processing by the operation is performed. An information processing program according to claim 10.

  According to a twelfth aspect of the present invention, a first storage means for storing a series of operations, area information indicating an area that does not change and information in the area are stored for a screen before or after performing the operation. A second storage unit, an execution unit that executes an operation stored in the first storage unit, and an area that should not change using the area information from a screen before or after execution by the execution unit. If the information does not match the information stored in the second storage means, the information processing apparatus has an error processing means for performing error processing.

  According to a thirteenth aspect of the present invention, a first storage means for storing a series of operations, area information indicating an area which does not change, and information in the area are stored for a screen before or after performing the operation. On a computer having a second storage means, an execution step of executing an operation stored in the first storage means, and using the area information from a screen before or after execution by the execution step, An information processing method including an error processing step of performing error processing when information in an area that should not change and information stored in the second storage unit do not match.

  The invention according to claim 14 causes the computer to function as error processing means for performing error processing when the information on the screen before or after the operation does not match the information in the area where the screen does not change. Information processing program.

  The invention according to claim 15 is an information processing apparatus including an error processing unit that performs error processing when information on a screen before or after performing an operation does not match information in an area where the screen does not change. .

  The invention according to claim 16, wherein, when information on a screen before or after performing an operation on a computer does not match information in an area where the screen does not change, an error processing step for performing an error processing is performed. Processing method.

  According to the information processing program of the first aspect, error processing can be performed when an operation different from the operation desired by the user is likely to be performed or when the operation is performed.

  According to the information processing program of the second aspect, the error processing can be performed when an operation different from the operation desired by the user is likely to be performed or is performed using the image in the area.

  According to the information processing program of the third aspect, the error processing can be performed using the characteristic information in the area when an operation different from the operation desired by the user is likely to be performed or is performed.

  According to the information processing program of the fourth aspect, region information indicating an unchanged region and information in the region are extracted from the screen before executing the operation at a different time from the screen before executing the operation. be able to.

  According to the information processing program of the fifth aspect, region information indicating an unchanged region and information in the region are extracted from the screen after performing the operation at a different time from the screen after performing the operation. be able to.

  According to the information processing program of the sixth aspect, it is possible to extract, from the screen before performing the operation and the screen after performing the operation, the region information indicating the region that has not changed and the information in the region. .

  According to the information processing program of the seventh aspect, when the internal state after the operation does not match the internal state stored in advance, error processing can be performed.

  According to the information processing program of the eighth aspect, a process of detecting an error can be performed before the first operation is performed.

  According to the information processing program of the ninth aspect, a process of detecting an error can be performed before each operation is executed.

  According to the information processing program of the tenth aspect, a process of detecting an error can be performed after the last operation is performed and before the data storage process is performed by the operation.

  According to the information processing program of the eleventh aspect, a process of detecting an error can be performed after each operation is performed and before the data storage process is performed by the operation.

  According to the information processing apparatus of the twelfth aspect, error processing can be performed when an operation different from the operation desired by the user is likely to be performed or when the operation is performed.

  According to the information processing method of the thirteenth aspect, error processing can be performed when an operation different from the operation desired by the user is likely to be performed or when the operation is performed.

  According to the information processing program of the fourteenth aspect, when an operation different from the operation desired by the user is likely to be performed, or when the operation is performed, error processing can be performed.

  According to the information processing apparatus of claim 15, it is possible to perform error processing when an operation different from the operation desired by the user is likely to be performed or when the operation is performed.

  According to the information processing method of the sixteenth aspect, error processing can be performed when an operation different from the operation desired by the user is likely to be performed or when the operation is performed.

FIG. 2 is a conceptual module configuration diagram illustrating a configuration example according to the present embodiment. FIG. 2 is an explanatory diagram illustrating a system configuration example using the present embodiment. It is an explanatory view showing a processing example by automatic operation. It is an explanatory view showing a processing example by automatic operation. It is an explanatory view showing a processing example by automatic operation. It is an explanatory view showing a processing example by automatic operation. 5 is a flowchart illustrating a processing example according to the exemplary embodiment; 5 is a flowchart illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating an example of a data structure of a verification area table. FIG. 9 is an explanatory diagram illustrating an example of a data structure of a verification area table. FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; 5 is a flowchart illustrating a processing example according to the exemplary embodiment; It is explanatory drawing which shows the example of a data structure of an operation pre / post screen table. 5 is a flowchart illustrating a processing example according to the exemplary embodiment; 5 is a flowchart illustrating a processing example according to the exemplary embodiment; FIG. 9 is an explanatory diagram illustrating a processing example according to the exemplary embodiment; FIG. 4 is an explanatory diagram illustrating an example of a data structure of a table. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer that implements the exemplary embodiment.

Hereinafter, an example of a preferred embodiment for realizing the present invention will be described with reference to the drawings.
FIG. 1 is a conceptual module configuration diagram of a configuration example according to the present embodiment.
Note that a module generally refers to a component such as software (computer program) and hardware that can be logically separated. Therefore, the module in the present embodiment indicates not only a module in a computer program but also a module in a hardware configuration. Therefore, the present embodiment is directed to a computer program for causing these modules to function as a program (a program for causing a computer to execute each procedure, a program for causing a computer to function as each means, ), And a description of a system and a method. However, for the sake of explanation, “store”, “store”, and equivalent words are used. However, when the embodiment is a computer program, these words are stored in a storage device or stored. This means that control is performed so as to be stored in the device. Further, the modules may correspond to the functions on a one-to-one basis, but in implementation, one module may be configured with one program, a plurality of modules may be configured with one program, and conversely, one module may be configured. May be composed of a plurality of programs. Also, a plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers in a distributed or parallel environment. Note that one module may include another module. Hereinafter, the term “connection” is used not only for a physical connection but also for a logical connection (data transmission / reception, instruction, reference relationship between data, login, etc.). "Predetermined" means that the process is determined before the process of interest, not only before the process according to the present embodiment starts, but also after the process according to the present embodiment starts. Also, before the target processing, it is used in accordance with the situation / state at that time or with the intention of being determined according to the situation / state up to that time. When there are a plurality of "predetermined values", the values may be different from each other, or two or more values (including all values, of course) may be the same. In addition, the expression “if it is A, do B” is used to mean “determine whether or not it is A, and if it is A, do B”. However, this does not apply to cases where it is not necessary to determine whether or not it is A. Further, when an object is enumerated such as “A, B, C” or the like, the enumeration is an example unless otherwise specified, and includes a case where only one of them is selected (for example, only A).
In addition, a system or an apparatus is configured by connecting a plurality of computers, hardware, apparatuses, and the like by communication means such as a network (including one-to-one correspondence communication connection), and one computer, hardware, and apparatus. Etc. are also included. “Apparatus” and “system” are used as synonymous terms. Of course, the “system” does not include anything that is merely a social “mechanism” (social system) that is an artificial arrangement.
In addition, for each process performed by each module or when a plurality of processes are performed in a module, target information is read from the storage device, and after performing the process, the processing result is written to the storage device. is there. Therefore, description of reading from the storage device before processing and writing to the storage device after processing may be omitted. Note that the storage device here may include a hard disk, a RAM (Random Access Memory), an external storage medium, a storage device via a communication line, a register in a CPU (Central Processing Unit), and the like.

  The information processing apparatus 100 according to the present embodiment is for automating operations, and as shown in the example of FIG. 1, an automatic operation module 105, a display control module 125, a verification area storage module 130, an area extraction module 135 , An error processing module 140. For example, the above-mentioned RPA corresponds. It is incorporated in the RPA to detect errors before or after the execution of the RPA. Further, in addition to the incorporation into the RPA, it may be incorporated into an RDA (Robotic Desktop Automation). Hereinafter, in the case of illustration, RPA is mainly used.

The automatic operation module 105 includes an operation reception module 110, an operation storage module 115, and an operation execution module 120. The automatic operation module 105 may be, for example, RPA or the like.
The operation reception module 110 is connected to the operation storage module 115 and the operation execution module 120. The operation receiving module 110 receives a user operation. For example, as described above, in addition to the operation using the mouse and keyboard on the screen, the operation using the finger and the pen on the touch panel, the user's operation using a camera, a microphone, and the like (including gaze, gesture, voice, and the like) May be accepted.

The operation storage module 115 is connected to the operation reception module 110 and the operation execution module 120. The operation storage module 115 stores a series of operations. For example, in RPA and the like, as a series of operations, specifically, an operation type (copy, click, etc.), coordinates at which the operation is performed, and the like are stored in the order of operation.
The operation execution module 120 is connected to the automatic operation module 105, the operation storage module 115, and the display control module 125. The operation execution module 120 executes an operation received by the operation reception module 110 or an operation (a series of operations) stored in the operation storage module 115. Executing the operation stored in the operation storage module 115 is a process in RPA or the like.

  The display control module 125 is connected to the operation execution module 120, the verification area storage module 130, the area extraction module 135, and the error processing module 140 of the automatic operation module 105. The display control module 125 controls display on a screen of a display device such as a liquid crystal display. For example, an execution result by the operation execution module 120, a message indicating that the error is caused by the error processing module 140 (or a message indicating that an error is likely to occur (or is highly likely to have occurred), etc.) are displayed. In addition to the display, the message to the user may be presented by audio output from a speaker, tactile sensation using a tactile device, or the like. In addition, as the error processing, the automatic processing may be interrupted (or stopped), the confirmation by the user may be prompted, or the like.

The verification area storage module 130 is connected to the display control module 125, the area extraction module 135, and the error processing module 140. The verification area storage module 130 stores area information indicating an area that does not change and information in the area for a screen before or after performing an operation.
Here, the “region information” may be information indicating a region that does not change before and after performing the operation. As a specific example, if the area is a rectangle, the information may be information indicating the rectangle (for example, information such as upper left coordinates, width, and height of the rectangle), or if the area is a polygon. May be information indicating the polygon or mask information. Note that the mask information is configured by information indicating the inside and outside of the region, and an image in the region can be extracted by a mask process using a logical product (AND) of the mask information and a target image (screen image).
Further, an image in the area may be used as the “information in the area”.
As the “information in the area”, characteristic information (also referred to as data characteristic) in the area may be used.
Here, the information itself or the attribute of the information displayed on the screen corresponds to the “characteristic information in the area”. For example, if a character string is displayed on the screen, the character string, the attributes of the character string (character types such as alphabets and numbers, the number of characters, the range of characters (if numeric, the range of numeric values, etc.)), etc. is there.

The area extraction module 135 is connected to the display control module 125, the verification area storage module 130, and the error processing module 140. The area extraction module 135 extracts area information indicating an unchanged area and information in the area from a screen before executing the operation and a screen before executing the operation at a different time from the screen before executing the operation. Control is performed so as to be stored in 130.
The target operation is performed repeatedly. Here, “another time point” refers to an operation at a different number of times after a plurality of operations are performed. For example, when the third operation is performed, it means the first or second operation.
Specifically, when the following operation is performed,
(1) First automatic operation: screen A-1 → operation A → screen B-1 → operation B → screen C-1
(2) Second automatic operation: screen A-2 → operation A → screen B-2 → operation B → screen C-2
(3) Third automatic operation: screen A-3 → operation A → screen B-3 → operation B → screen C-3
In the operation A, the screen A-1 and the screen A-2, and the screen A-1 and the screen A- as "" screen before performing the operation "and" screen before performing the operation at another time "" 3, screen A-2 and screen A-3, combination of screen A-1 and screen A-2 and screen A-3, and in operation B, screen B-1 and screen B-2, screen B-1 and screen B-3, the combination of the screen B-2 and the screen B-3, and the combination of the screen B-1 and the screen B-2 and the screen B-3. The comparison target may be two screens or three or more screens.

Further, the region extraction module 135 extracts region information indicating a region that has not changed and information in the region from the screen after performing the operation at a different time from the screen after performing the operation, and extracts the verification region. You may make it control so that it may be memorize | stored in the storage module 130.
Here, in the above example, the “screen after execution of the operation” and “the screen after execution of the operation at another time” are described as “screen B-1 and screen B-2, B-1 and screen B-3, screen B-2 and screen B-3, screen B-1 and screen B-2 and screen B-3, and in operation B, screen C-1 and screen C-2 , Screen C-1 and screen C-3, screen C-2 and screen C-3, and screen C-1 and screen C-2 and screen C-3. The comparison target may be two screens or three or more screens.

The area extraction module 135 extracts area information indicating an unchanged area and information in the area from the screen before the operation is performed and the screen after the operation is performed, and the verification area storage module 130 May be controlled to be stored.
In this case, in the above-described example, the screen A-1, the screen B-1, and the screen A-2 in the operation A are referred to as “the screen before the operation is performed” and the “screen after the operation is performed”. And the screen B-2, and the combination of the screen A-3 and the screen B-3. In the operation B, a combination of the screen B-1 and the screen C-1, the screen B-2 and the screen C-2, and the screen B-3 and the screen C-3 correspond. The comparison targets are two screens before and after the operation.

  The error processing module 140 is connected to the display control module 125, the verification area storage module 130, and the area extraction module 135. The error processing module 140 performs error processing when the information on the screen before or after the operation does not match the information in the area where the screen does not change. Specifically, the error processing module 140 uses the area information from the screen before or after execution by the operation execution module 120, and uses information in an area that should not change and information stored in the verification area storage module 130. If does not match, error processing is performed.

Further, the verification area storage module 130 may store an internal state after the operation is performed.
Here, for example, information copied to an internal buffer corresponds to the “internal state”. Specifically, by pressing the "C" key while pressing the "Ctrl" key, which is a shortcut key, the information selected at that time is copied to the internal buffer, and the copied information is applicable. . Of course, not only the shortcut key but also a “copy” operation by a mouse operation may be used. Of course, the information in this internal buffer can be extracted by the area extraction module 135.
Then, the error processing module 140 may perform error processing when the internal state after the operation is performed and the internal state stored in the verification area storage module 130 do not match.

In addition, the error processing module 140 may perform a process of detecting an error before at least the first operation of the operations executed by the operation execution module 120 is executed.
Here, “the process of detecting an error” is “stored in the verification area storage module 130 and information in an area that should not change using area information from a screen before or after execution by the operation execution module 120. This is a process of determining whether or not the information matches.
The error processing module 140 may perform a process of detecting an error before each operation executed by the operation execution module 120 is executed.

The error processing module 140 performs a process of detecting an error after at least the last operation of the operations performed by the operation execution module 120 has been performed and before data storage processing by the operation has been performed. It may be performed.
Further, the error processing module 140 may perform a process of detecting an error after each operation performed by the operation execution module 120 is performed and before data storage processing is performed by the operation. Good.

FIG. 2 is an explanatory diagram showing a system configuration example using the present embodiment.
FIG. 2A shows an example in which the information processing apparatus 100 is built in a personal computer 200 as a stand-alone system. The personal computer 200 is used by the user 250 and has the information processing device 100. It supports single application and multiple applications, and mainly supports RDA (which does not exclude RPA). For example, by utilizing functions that realize semi-automation of work such as macro functions, repetitive work to regularly organize a large amount of information, routine work that is performed regularly according to a predetermined process (routine work) Automation for applications such as.

FIG. 2B shows an example of a system configuration in which the user terminal 210 uses the information processing device 100 via the communication line 290.
The user terminal 210 (user terminal 210A, user terminal 210B, user terminal 210C) is used by the user 250 (user 250A, user 250B, user 250C). The information processing device 100, the user terminal 210A, the user terminal 210B, and the user terminal 210C are connected via a communication line 290, respectively. The communication line 290 may be wireless, wired, or a combination thereof, and may be, for example, the Internet or an intranet as a communication infrastructure. Further, the function of the information processing apparatus 100 may be realized as a cloud service. It mainly supports RPA (which does not exclude RDA) for single applications and multiple applications. For example, a simple task that requires cooperation of a plurality of applications may be performed, and for that purpose, a function such as a rule engine, image recognition, coordinate acquisition, and business flow management may be installed.

A processing example of the automatic operation by the RPA will be described with reference to FIGS. 3 and 4 show examples of normal operation, and FIGS. 5 and 6 show examples of abnormal operation.
FIG. 3 is an explanatory diagram illustrating an example of processing by an automatic operation. It shows an example of the state before the execution of the operation.
The screen 300 displays a Web page 330 on the left and a spreadsheet screen 340 on the right. Then, a table 310 is displayed on the Web page 330, and a table 320 is displayed on the spreadsheet screen 340.
As the processing contents of the user 250, it is assumed that an operation of copying a price in the table 310 on the left to a price column in the table 320 on the right is performed every day. This is the target of RPA.
In the table 310, the type of the fruit is described in the first row, and the price of the fruit is described in the second row. Specifically, the table 310 is a table having two rows and three columns, and indicates that "apples: 100 yen", "orange: 80 yen", and "grape: 500 yen" for each column.
In the table 320, the type of the fruit is described in the first column, and the price of the fruit is described in the second column. Specifically, the table 320 is a table with three rows and two columns, and for each row, “apple: (blank) circle”, “orange: (blank) circle”, and “grape: (blank) circle” Is shown.

FIG. 4 is an explanatory diagram illustrating an example of processing by an automatic operation. It shows an example of a state after the execution of the operation.
In the state shown in the example of FIG. 3, when the price of the table 310 is to be copied to the price column of the table 320, the contents to be automatically operated are as follows.
By the operation of the user 250, the value (price) in the second row of the table 310 is selected, and the selected content is copied (copied) in the second column of the table 320.
That is, “100 yen” in the second row and the first column of the table 310 is selected, the selected content is copied to the first row and the second column of the table 320, and “80 yen” in the second row and the second column of the table 310 is changed. Select and copy the selected content to the second row and second column of the table 320, select "500 yen" in the second row and third column of the table 310, and copy the selected content to the third row and second column of the table 320 Will be.

In this case, the GUI storage contents in the operation storage module 115 for automatic operation are as follows. The line with the number enclosed by [] becomes the storage content.
(1) Start [21] Open the web page 330.
[22] The spreadsheet software screen 340 is opened.
(3) Start editing data.
[41] Left click. The location is to the left of the first character in the second row, first column of table 310.
[42] Left-click while pressing the Shift key. The location is to the right of the last character in the second row, first column of table 310.
[43] Press C while holding down the Ctrl key.
[44] Left click. The location is the first row and second column of the table 320.
[45] Press V while holding down the Ctrl key.
Data editing is repeated by repeating steps [41] to [45] while changing the location.
(5) Data editing is completed.
[61] The data entered in the table 320 is saved.
[62] The Web page 330 is closed.
[63] The spreadsheet software screen 340 is closed.
(7) End When the automatic operation by the RPA is performed in accordance with the contents of the GUI storage, the state shown in the example of FIG. 3 is changed to the state shown in the example of FIG. That is, the prices “100 yen”, “80 yen”, and “500 yen” in Table 310 are copied to Table 320.

FIG. 5 is an explanatory diagram illustrating an example of processing by an automatic operation. It shows an example of the state before the execution of the operation.
The screen 500 displays a Web page 530 on the left and a spreadsheet software screen 540 on the right. A table 510 is displayed on the Web page 530, and a table 520 is displayed on the spreadsheet software screen 540.
Table 510 is a table having two rows and three columns, and indicates that "apple: 100 yen", "orange: 80 yen", and "grape: 500 yen" for each column.
However, unlike the table 310 shown in the example of FIG. 3, the first row and the second row are switched, the first row shows the price, and the second row shows the kind of fruit.

FIG. 6 is an explanatory diagram illustrating an example of processing by an automatic operation. It shows an example of a state after the execution of the operation.
In the state shown in the example of FIG. 5, when processing is performed by RPA (conventional RPA) not incorporating this embodiment, the value (product name) in the second row of table 510 is replaced by the second row in table 520. Copied to column.
That is, “apple” in the second row and the first column of the table 510 is copied to the first row and the second column of the table 520, and “orange” in the second row and the second column of the table 510 is copied in the second row and the second row of the table 520. The “grape” in the second row and third column of table 510 is copied to the third row and second column of table 520.
This is because, despite the fact that the inside of the table 510 is different from the usual, the coordinates are copied according to the coordinates (xy coordinates on the screen, coordinates indicated by the matrix in the table (cells in the first row, first column, etc.)). Due to the processing, the original price (the first row in the table 510) could not be copied to the second column of the table 520, which is different from the result desired by the user. ing. According to the present embodiment, in such a case, error processing is performed.

FIG. 7 is a flowchart illustrating a processing example according to the present embodiment.
In step S702, it is determined whether or not advance preparations for the RPA process have been completed. If the preparations have been completed, the process proceeds to step S704. Otherwise, the process waits for completion. For example, the contents of an automatic operation of the RPA (an example of “a series of operations”) are stored in the operation storage module 115, the region information indicating an unchanging region on the screen and the information in the region are stored in the operation execution module 120. Confirm that it is stored in.
In step S704, preliminary verification is performed. Detailed processing of step S704 will be described later using a flowchart illustrated in the example of FIG.
In step S706, an automatic operation is performed. Specifically, an RPA process is performed.
In step S708, post-verification is performed. Detailed processing of step S708 will be described later using a flowchart illustrated in the example of FIG.

FIG. 8 is a flowchart illustrating a processing example according to the present embodiment.
In step S802, the image (A) in the verification area is extracted from the current screen. To extract the image (A) in the verification area, for example, the verification area table 900 is used. The verification area table 900 is stored in the verification area storage module 130. FIG. 9 is an explanatory diagram showing an example of the data structure of the verification area table 900. The verification region table 900 has an RPA-ID column 910, a verification region image column 920, and a verification region mask image column 930. The RPA-ID column 910 stores information (RPA-ID: IDentification) for uniquely identifying an RPA in the present embodiment. The verification area image field 920 stores the verification area image in the RPA. The verification region mask image column 930 stores a verification region mask image (an example of “region information indicating a region that does not change”) for extracting the verification region image.
That is, in step S802, the image (A) may be extracted from the current screen using the verification area mask image field 930 of the verification area table 900 corresponding to the RPA that has started processing. More specifically, a snapshot image of the current screen is obtained, and the logical product of the snapshot image and the verification area mask image is performed to obtain the image (A) (“information in the area that should not change”). Example) is extracted. If the mask image is, for example, an image represented by “1” and “0”, “1” indicates a portion to be extracted, and “0” indicates a portion not to be extracted, the logical product By performing the processing, it is possible to extract only the image in the area of “1”.

  In step S804, an image (B) of the verification area corresponding to the RPA is extracted. For example, a verification area image (an example of “information stored in the verification area storage module 130 (information in an area that does not change)”) in the verification area image column 920 may be extracted from the verification area table 900.

In step S806, the image (A) is compared with the image (B). Conventional processing may be used for the image comparison processing. For example, an exclusive OR (EOR) process may be performed between the two.
In step S808, it is determined whether or not the image (A) and the image (B) do not match. If they do not match, the process proceeds to step S810. Otherwise (the case of matching), the process ends (step S808). S899). In the case of a mismatch, it indicates that some change has occurred in the area that should not change, and error processing should be performed.
In step S810, error processing is performed. For example, an error message indicating that there is a high possibility that an error may occur may be displayed in step S704. In step S708, an error message indicating that there is a high possibility that an error has occurred may be displayed. Further, an image of a portion that has become inconsistent by the comparison in step S806 (the image after the above-described EOR processing) may be displayed. If the user 250 determines that no problem occurs in the process using the RPA even if there is a mismatch, the process may be allowed to continue. For example, when an inconsistency occurs in an area other than the table 310 (for example, a display area such as a date) in the Web page 330, a “continue processing” button may be selected and the processing may be continued.

Note that the verification area table 1000 may be used instead of the verification area table 900. FIG. 10 is an explanatory diagram showing an example of the data structure of the verification area table 1000.
The verification area table 1000 has an RPA-ID column 1010, a pre / post-change column 1015, a verification region image column 1020, and a verification region mask image column 1030. The RPA-ID column 1010 stores the RPA-ID. The pre / post column 1015 stores information (flag) indicating whether it is for pre-verification or post-verification. The verification area image field 1020 stores a verification area image. The verification area mask image field 1030 stores a verification area mask image. The verification area table 1000 is obtained by adding a pre / post column 1015 to the verification area table 900. In the verification in step S704, the one in which the pre / post column 1015 is a pre-verification flag is used, and in the verification in step S708, the pre / post column 1015 is a post-verification flag. May be used.

An example of “region information indicating a region that does not change and information in the region for a screen before or after performing an operation” stored in the verification region storage module 130 will be described using the examples of FIGS. 11 to 22. .
FIG. 11 is an explanatory diagram illustrating a processing example according to the present embodiment. 9 shows an example in the verification area image column 920 of the verification area table 900.
Screen 1100 shows the state before the RPA process. That is, it is assumed that, when the web page 1130 is opened, only the price in the table 1110 is changed, and other character strings (names of tables and fruits), graphic frames, blank portions, and the like are not changed.
Further, it is assumed that when the spreadsheet software screen 1140 is opened, the table 1120 is not changed.
A screen 1100 shows an invariable area 1150 in a state where a Web page 1130, a table 1110, a spreadsheet software screen 1140, and a table 1120 are displayed. The invariable area 1150 is a shaded portion.
Further, an area other than the invariable area 1150 (an area other than the hatched area) in the screen 1100 is an area to be changed.

12 and 13 are explanatory diagrams illustrating a processing example according to the present embodiment.
The example of FIG. 12 shows the result of extracting the verification area image 1200 from the screen 1100 using the invariable area 1150 shown in the example of FIG. The second row of Table 1110 is blank.
The verification area image 1200 (the image in the invariable area 1150) becomes the verification area image in the verification area image field 920 of the verification area table 900 (or the verification area image field 1020 of the verification area table 1000). The masked image becomes the mask image in the verification region mask image column 930 of the verification region table 900 (or the verification region mask image column 1030 of the verification region table 1000). Specifically, the mask image is an image in which the invariable area 1150 (shaded portion) is set to “1” and the other portions are set to “0”.
The change area image 1310 in the original screen 1300 shown in the example of FIG. 13 is an image other than the verification area image 1200.
In this case, in step S704 in the flowchart shown in the example of FIG. 7, the verification area image 1200 shown in the example of FIG. 12 is compared with the snapshot image of the screen 500 shown in the example of FIG. Since the images in the first column of the table 510 are different, error processing is performed.

FIG. 14 is an explanatory diagram illustrating a processing example according to the present embodiment. 9 shows an example in the verification area image column 920 of the verification area table 900.
Screen 1400 shows the state after the RPA process. That is, the result of the data editing (operation of copying the price data in Table 1410 into the second column of Table 1420) by the RPA process is shown.
On the screen 1400, a Web page 1430, a table 1410, a spreadsheet software screen 1440, and a table 1420 are displayed, showing the invariable area 1450 in a state after the RPA processing. Note that the invariable region 1450 is a hatched portion.
An area other than the invariable area 1450 (an area other than the hatched area) in the screen 1400 is an area to be changed.

FIG. 15 and FIG. 16 are explanatory diagrams showing processing examples according to the present embodiment.
The example of FIG. 15 shows the result of extracting the verification area image 1500 from the screen 1400 using the invariable area 1450 shown in the example of FIG. The second row of the table 1410 and the second column of the table 1420 are blank.
The verification area image 1500 (the image in the invariable area 1450) becomes the verification area image in the verification area image field 920 of the verification area table 900 (or the verification area image field 1020 of the verification area table 1000). The masked image becomes the mask image in the verification region mask image column 930 of the verification region table 900 (or the verification region mask image column 1030 of the verification region table 1000).
Then, the changed area image 1610 and the changed area image 1620 in the original screen 1600 shown in the example of FIG. 16 are images other than the verification area image 1500.
In this case, in step S708 in the flowchart illustrated in the example of FIG. 7, the verification area image 1500 illustrated in the example of FIG. 15 is compared with the snapshot image of the screen 500 illustrated in the example of FIG. Since the images in the area in the first column of the table 510 are different, error processing is performed.

The flowchart illustrated in the example of FIG. 17 illustrates a process of collecting a snapshot image.
FIG. 17A is a flowchart illustrating a processing example according to the present embodiment. More specifically, in the setting of the RPA, snapshot images of the screen before the setting (before the operation by the user 250) and the screen after the setting (after the operation by the user 250) are collected.
In step S1702a, a snapshot image of the screen before the automatic operation setting is generated.
In step S1704a, a snapshot image of the screen after the automatic operation setting is generated.

FIG. 17B is a flowchart illustrating a processing example according to the present exemplary embodiment. Specifically, after the RPA is set, snapshot images of screens before and after the automatic operation are collected.
In step S1702b, a snapshot image of the screen before the execution of the automatic operation is generated.
In step S1704b, a snapshot image of the screen after the execution of the automatic operation is generated.

FIG. 18 is an explanatory diagram showing an example of the data structure of the operation pre / post screen table 1800. The operation pre / post screen table 1800 stores a snapshot image generated according to the flowchart shown in the example of FIG.
The operation pre / post screen table 1800 has an RPA-ID column 1810, a date / time column 1820, a pre / post column 1830, and a snapshot image column 1840. The RPA-ID column 1810 stores the RPA-ID. The date and time column 1820 stores the date and time when the snapshot image was generated. The pre / post column 1830 stores information (flag) indicating whether the screen is an operation pre-screen (step S1702a, step S1702b) or an operation post-screen (step S1704a, step S1704b). The snapshot image column 1840 stores a snapshot image.

FIG. 19 is a flowchart illustrating a processing example according to the present embodiment. "Extract area information indicating an unchanged area and information in the area from the screen before performing the operation at a different time from the screen before performing the operation", "The screen after performing the operation Area information indicating an area that has not changed and information within the area are extracted from the screen after the operation is performed at another time. "
In step S1902, pre- and post-snapshot images of the target RPA-ID are extracted. Before and after each, a plurality of snapshot images are extracted.
In step S1904, a matching area between the previous snapshot images is extracted, and a mask image is generated. For example, in order to extract a coincident region, the exclusive OR operation between the two images may be negated (NOT (EOR (snapshot image A, snapshot image B))).
In step S1906, a matching area between the subsequent snapshot images is extracted, and a mask image is generated.
In step S1908, the image is registered as a verification area image. More specifically, the information may be registered in the verification area table 900 or the verification area table 1000. More specifically, the matching region may be stored in the verification region image column 920 (verification region image column 1020), and the mask image may be stored in the verification region mask image column 930 (verification region mask image column 1030).

FIG. 20 is a flowchart illustrating a processing example according to the present embodiment. 9 illustrates a processing example of “extracting area information indicating an unchanged area and information in the area from a screen before performing the operation and a screen after performing the operation”.
In step S2002, the snapshot images before and after the target RPA-ID are extracted.
In step S2004, a region where the previous snapshot image matches the subsequent snapshot image is extracted, and a mask image is generated.
In step S2006, the image is registered as a verification area image. More specifically, the information may be registered in the verification area table 900 or the verification area table 1000.
The verification region image and the mask image may be generated by further performing a logical product process on the verification region image generated by the processing of the flowchart illustrated in the example of FIG. 20 with the verification region image in advance.

Next, an example in which “characteristic information in an area is used as information in the area” in the verification area storage module 130 will be described using the examples in FIGS. 21 and 22.
FIG. 21 is an explanatory diagram illustrating a processing example according to the present embodiment.
An example is shown in which a web page 2130, a table 2110, a spreadsheet software screen 2140, and a table 2120 are displayed on the screen 2100, and identification information (partial area ID) is assigned to each partial area.
That is, “A1-0” is set in an area other than the table 2110 of the Web page 2130, “A1-1-1” to “A1-2-3” is set in each cell in the table 2110, and the table is set to a position other than the table 2120 of the spreadsheet software screen 2140 , The IDs of “A2-0” are assigned to the cells in Table 2120, and the IDs of “A2-1-1” to “A2-3-2” are assigned to the cells in Table 2120.

FIG. 22 is an explanatory diagram illustrating an example of the data structure of the table 2200. The table 2200 is stored in the verification area storage module 130.
The table 2200 has a partial area ID column 2210, a data characteristic column 2220 before data editing, and a data characteristic column 2230 at a point after data editing. The partial area ID column 2210 stores information (partial area ID) for uniquely identifying a partial area in the present embodiment. The data characteristic column before data editing 2220 stores the data characteristic before the data editing in the partial area. The data characteristic at time point after data editing column 2230 stores the data characteristic at the time point after data editing in the partial area.
For example, regarding the partial area “A1-0”, the data characteristic before the data editing is “character string“ Table 1 ””, and the data characteristic after the data editing is “character string“ Table 1 ””. For the partial area “A1-2-1”, the data characteristic before the data editing is “numerical string + character string“ circle ”. The number is 200 or less, and the data characteristic at the time after data editing is “numerical string + character string“ yen ”. The number is 200 or less ", and for the partial area" A2-1-2 ", the data characteristic before the data editing is" blank ", and the data characteristic after the data editing is" numerical ". Column + character string "Yen". The number is 200 or less. "
Here, for the derivation of “the number is 200 or less”, a statistical process (a process of obtaining a maximum value, a minimum value, an average value, a median value, a mode value, etc.) may be performed from a plurality of operation results.

On the screen at the time of starting the automatic operation or at the time of ending the automatic operation, the characteristics of the data of each partial area are registered in the table 2200.
Before or after the execution of the automatic operation registered in the operation storage module 115 at the time of execution of the automatic operation, for each partial area, the data characteristics registered in the table 2200 of the partial area and the actual corresponding partial area The data characteristic (data characteristic of the character in the partial area) is compared.
Further, the comparison may be performed not for before or after execution of the automatic operation but for each process of each step in the automatic operation. Specifically, at the time of execution of the automatic operation, before or after the processing of each step in the automatic operation registered in the operation storage module 115, the data characteristic of the partial area to be processed (character of the character in the partial area) Data characteristic) and the data characteristic registered in the table 2200 are compared.
If they do not match, an error is determined. In this case, for example, an error message is displayed and the automatic operation process is stopped. In addition, as the content of the error display, it may be possible to point out a non-matching partial area, to display the registered content in the table 2200 and the content of the partial area which is the actual operation target, or the like. Of course, if they match, the automatic operation is continued.

  Further, a partial area where the contents of the screen is not changed (for example, A1-0) and a partial area where the contents of the screen are changed (for example, A2-1-2) are distinguished. The data characteristics of the region may be registered, or the data characteristics of the partial region may be registered only for the partial region where the contents of the screen are changed. As a result, the amount of data registered in the table 2200 can be reduced. For example, it is sufficient to extract only a partial area where the contents of the screen are not changed, and confirm that the data characteristics are the same as those of the table 2200 and that the contents are not changed before and after the operation. In addition, it is only necessary to extract only the partial area where the contents of the screen are changed, and confirm that the data characteristics are the same as those of the table 2200 before and after the operation.

The internal state of the personal computer 200 may be registered as the data characteristic in the table 2200. That is, the internal state after the automatic operation (or after the processing of each step in the automatic operation) is registered in the table 2200. As described above, specifically, the internal state includes information copied to the internal buffer.
After each process of the automatic operation is performed, the internal data is extracted and compared with the data in the table 2200 to detect an error.

In the example of FIG. 7, the verification process is performed before and after the automatic operation in step S706, as shown in steps S704 and S708. Even in the case of “using an image in the area as information in the area”, as shown in the description of the example of FIG. 22, the verification processing is performed before and after the processing of each step configuring the automatic operation. It may be. In this case, an error is not missed. However, as compared with the case where the verification process is performed before and after the automatic operation, the registration work of “the area information indicating the area that does not change and the information in the area” increases, and the execution time of the automatic operation also increases. .
Therefore, in order to reduce the number of verification processes as compared with the case where the verification process is performed before and after each step, as shown in the example of FIG. 7, the verification process is performed before and after the execution of the automatic operation.
The “before performing the automatic operation” may be, for example, “before starting necessary software (Web browser, spreadsheet software in the above-described example) and starting the automatic operation for data editing”. Further, "after the execution of the automatic operation" may be, for example, "before the automatic operation of the data editing is completed and before the data written in the software is stored".
Further, in order to reduce the verification processing, the verification processing may be performed only “after the execution of the automatic operation”.
If the detection processing is performed “before performing the automatic operation”, an error can be detected and the automatic processing can be stopped before the data editing processing, so that an abnormal start of the data editing operation can be prevented.
When the detection processing is performed “after the execution of the automatic operation”, an error can be detected before the data is stored and the automatic processing can be stopped, so that the adverse effect on the storage contents of the software at the data storage destination can be reduced.

  The hardware configuration of a computer that executes the program according to the present embodiment is a general computer as illustrated in FIG. 23, and specifically, a personal computer, a computer that can be a server, or the like. That is, as a specific example, the CPU 2301 is used as a processing unit (arithmetic unit), and the RAM 2302, the ROM 2303, and the HD 2304 are used as storage devices. As the HD 2304, for example, a hard disk and an SSD (Solid State Drive) may be used. A CPU 2301 for executing programs such as an operation reception module 110, an operation execution module 120, a display control module 125, an area extraction module 135, and an error processing module 140; a RAM 2302 for storing the programs and data; ROM 2303 in which programs and the like are stored, HD2304 which is an auxiliary storage device (which may be a flash memory or the like) having functions as operation storage module 115, verification area storage module 130, and the like, keyboard, mouse, and touch A reception device 2306 for receiving data based on user operations (including operation, voice, line of sight, etc.) on a screen, microphone, camera (including a line-of-sight detection camera, etc.), and outputs from a CRT, liquid crystal display, speaker, etc. A location 2305, a communication line interface 2307 for connecting to a communication network such as a network interface card, and, and a bus 2308 for exchanging data by connecting them. A plurality of these computers may be connected to each other by a network.

Of the above-described embodiments, the ones according to the above-described embodiments are implemented by causing a system having the present hardware configuration to read a computer program which is software, and the software and hardware resources cooperate with each other. Is realized.
Note that the hardware configuration shown in FIG. 23 shows one configuration example, and the present embodiment is not limited to the configuration shown in FIG. 23, and is a configuration capable of executing the modules described in this embodiment. I just need. For example, some modules may be constituted by dedicated hardware (for example, an application-specific integrated circuit (ASIC) or the like), and some modules may be in an external system and connected by a communication line. 23. Further, a plurality of systems shown in FIG. 23 may be connected to each other by a communication line to operate in cooperation with each other. In particular, in addition to personal computers, portable information communication devices (including mobile phones, smartphones, mobile devices, wearable computers, etc.), information appliances, robots, copiers, fax machines, scanners, printers, and multifunction devices (scanners, printers, An image processing apparatus having at least two functions, such as a copying machine and a facsimile, may be incorporated.

  In the above-described embodiment, “the area information indicating the area that does not change and the information in the area for the screen before or after performing the operation” stored in the verification area storage module 130 (the verification area table 900, Although the example of the generation processing of the verification area table 1000 and the table 2200) has been described, when the user 250 or the like registers the contents of the automatic operation, the user 250 or the like extracts the unchanged area on the screen and stores the verification area. You may make it register in the module 130.

The above-described program may be provided by being stored in a recording medium, or the program may be provided by a communication unit. In this case, for example, the above-described program may be regarded as an invention of a “computer-readable recording medium on which the program is recorded”.
The “computer-readable recording medium on which the program is recorded” refers to a computer-readable recording medium on which the program is recorded, which is used for installing, executing, and distributing the program.
The recording medium is, for example, a digital versatile disk (DVD), which is a standard defined by the DVD Forum, such as "DVD-R, DVD-RW, DVD-RAM, etc.", and DVD + RW. Blu-ray discs such as "DVD + R, DVD + RW, etc." which are standards, compact discs (CDs) such as read-only memory (CD-ROM), CD recordable (CD-R), CD rewritable (CD-RW), etc. Blu-ray (registered trademark) Disc), magneto-optical disk (MO), flexible disk (FD), magnetic tape, hard disk, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM (registered trademark) )), Flash memory, random access memory (RAM) , SD (Secure Digital) memory card, and the like.
The whole or a part of the program may be recorded on the recording medium and stored or distributed. In addition, the communication may be, for example, a wired network used for a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, an intranet, an extranet, or a wireless communication. The data may be transmitted using a transmission medium such as a network or a combination thereof, or may be transmitted on a carrier wave.
Further, the above program may be a part or all of another program, or may be recorded on a recording medium together with a separate program. Further, the program may be divided and recorded on a plurality of recording media. Further, the program may be recorded in any form as long as it can be restored, such as compression or encryption.

REFERENCE SIGNS LIST 100 information processing device 105 automatic operation module 110 operation reception module 115 operation storage module 120 operation execution module 125 display control module 130 verification area storage module 135 area extraction module 140 error processing module 200 personal computer 210: User terminal 250: User 290: Communication line

Claims (16)

Computer
First storage means for storing a series of operations;
For the screen before or after performing the operation, area information indicating an area that does not change and second storage means storing information in the area,
Executing means for executing an operation stored in the first storage means;
Using the area information from the screen before or after execution by the execution means, if the information in the area that should not change and the information stored in the second storage means do not match, an error processing error is performed. Information processing program for functioning as processing means. Using information in the area as information in the area,
The information processing program according to claim 1. As the information in the area, using the characteristic information in the area,
The information processing program according to claim 1. From the screen before executing the operation and the screen before executing the operation at another time, the area information indicating the unchanged area and the information in the area are extracted and stored in the second storage means. The information processing program according to claim 1, further comprising control means for controlling. From the screen after performing the operation and the screen after performing the operation at another time, the region information indicating the unchanged region and the information in the region are extracted and stored in the second storage unit. The information processing program according to claim 1, further comprising control means for controlling. Control for extracting area information indicating an unchanged area and information in the area from a screen before the operation is performed and a screen after the operation is performed, and controlling the information to be stored in the second storage unit. The information processing program according to claim 1, further comprising: means. The second storage unit stores an internal state after performing the operation,
The error processing unit performs error processing when the internal state after performing the operation does not match the internal state stored in the second storage unit.
The information processing program according to claim 1. The error processing unit performs a process of detecting an error before at least a first operation of the operations performed by the execution unit is performed.
The information processing program according to claim 1. The error processing unit performs a process of detecting an error before each operation performed by the execution unit is performed.
An information processing program according to claim 8. The error processing unit performs a process of detecting an error after at least the last operation of the operations performed by the execution unit has been performed and before data storage processing by the operation is performed.
The information processing program according to claim 1. The error processing unit performs a process of detecting an error after each operation performed by the execution unit has been performed and before data storage processing by the operation is performed.
The information processing program according to claim 10. First storage means for storing a series of operations;
For the screen before or after performing the operation, area information indicating an area that does not change and second storage means storing information in the area,
Executing means for executing an operation stored in the first storage means;
Using the area information from the screen before or after execution by the execution means, if the information in the area that should not change and the information stored in the second storage means do not match, an error processing error is performed. An information processing apparatus having processing means. First storage means for storing a series of operations;
For a screen before or after performing an operation, on a computer having area information indicating an area that does not change and second storage means storing information in the area,
An execution step of executing an operation stored in the first storage means;
Using the area information from the screen before or after the execution in the execution step, if the information in the area that should not change and the information stored in the second storage unit do not match, an error processing error is performed. An information processing method that performs a processing step. Computer
An information processing program for functioning as error processing means for performing error processing when information on a screen before or after performing an operation does not match information in an area where the screen does not change. An information processing apparatus comprising: an error processing unit that performs error processing when information on a screen before or after performing an operation does not match information in an area where the screen does not change. On the computer,
An information processing method that performs an error processing step of performing error processing when information on a screen before or after performing an operation does not match information in an area where the screen does not change.

Images