JP2015076041A - Erroneous input detection device and erroneous input detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress risk of an erroneous input and to reduce time and effort required for confirmation work.SOLUTION: An erroneous input detection device and an erroneous input detection program which confirm whether an automatic operation device 91 has adequately performed automatic operation on an object 92 to be automated, includes: a result confirmation part 13 for detecting change on the object 92 to be automated before and after the automatic operation on the object 92 to be automated performed by the automatic operation device 91 and for determining whether the automatic operation has been adequately performed based on the detected change.

Description

  The present invention relates to an erroneous input detection apparatus and an erroneous input detection program for recording a user operation procedure in a format that can be automated by a computer in a system that automates a user operation on an interface displayed on a screen of a computer or the like.

  The automation software can automatically input a character string to the GUI instead of a human. Like humans, there is no mistaken keyboard input, but there is a lack of input characters or unintended characters due to high load or malfunction in any or all of the terminals, networks, servers, etc. that make up the target system. May be replaced (so-called garbled characters). In addition, even in an automatic operation for a GUI component that is operated by a mouse click such as a button, the click may not be recognized as intended depending on the environment or timing.

  In general, there is a possibility that input characters may be lost or garbled due to high load or malfunction on the target system regardless of either automatic execution by automation software or human operation. In particular, low-level automation that emulates text transmission and keyboard and mouse movements is susceptible.

  When a human is inputting, it is recognized that a character is not input as intended while visually inputting characters, and the input can be appropriately corrected. On the other hand, in the case of automatic execution by automation software, it is not possible to confirm on the spot whether or not the input has been made as intended, and there is a risk that incorrect contents are registered in the system.

  Although there are cases where errors can be detected by performing an operation of performing visual confirmation by humans after registration by automatic execution, it is inefficient. In addition, if it is assumed that the system is under heavy load, there is a method to suppress the occurrence of the problem by adjusting the character input speed to a slower speed. Is reduced and inefficient.

  When it is possible to acquire the internal state after automatic operation, it is possible to detect an erroneous input by comparing the expected state and the resulting state, but the internal state cannot be acquired (permitted If not, this method is not applicable. In that case, it is impossible to visually confirm the problem, and the problem is overlooked or the discovery of the problem is delayed.

“UWSC (Free version)” http: // www. uwsc. info / “RocketMouse Pro” http: // mojosoft. biz / products / rocketmousepro /

  An object of the present invention is to reduce the risk of erroneous input and reduce the time and labor required for confirmation work.

  In order to achieve the above object, the present invention is characterized in that a human-natural input operation that is performed while visually observing is performed even when a GUI button is clicked or a character string is input by automation software.

Specifically, the erroneous input detection device according to the present invention is:
A change detection unit for detecting a change in the automation target before and after an automatic operation on the automation target performed by the automatic operation device;
A determination unit that determines whether or not an automatic operation has been appropriately performed based on the change detected by the change detection unit;
Is provided.

In the erroneous input detection device according to the present invention,
The change detection unit images an operation part that performs an automatic operation in the screen display of the automation target, detects a change in the image before and after the automatic operation,
The determination unit determines that the automatic operation has been appropriately performed when the change in the image before and after the automatic operation has a difference determined by the automatic operation, and the difference in which the change in the image before and after the automatic operation is determined by the automatic operation It may be determined that the automatic operation has not been appropriately performed when the difference is different.

In the erroneous input detection device according to the present invention,
The change detection unit detects a change in a character string input to the automation target before and after an automatic operation,
The determination unit determines that the automatic operation is appropriately performed when the change in the character string detected by the change detection unit matches the character string input by the automatic operation device, and the character detected by the change detection unit. It may be determined that the automatic operation has not been appropriately performed when the change in the sequence does not match the character string input by the automatic operation device.

In the erroneous input detection device according to the present invention,
The change detection unit images the character string input by the automatic operation in the screen display to be automated, and also images the character string input by the automatic operation device,
The determination unit determines that the automatic operation is appropriately performed when the image of the character string imaged by the change detection unit matches, and the image of the character string imaged by the change detection unit does not match It may be determined that the automatic operation has not been properly performed.

Specifically, the erroneous input detection program according to the present invention is:
A change detecting step for detecting a change in the automation target before and after an automatic operation on the automation target performed by the automatic operation device;
A determination step of determining whether or not automatic operation has been appropriately performed based on the change detected in the change detection step;
Is an erroneous input detection program for causing a computer to execute.

In the erroneous input detection program according to the present invention,
In the change detection step, an operation part for performing an automatic operation in the screen display of the automation target is imaged, and a change in the image before and after the automatic operation is detected,
In the determination step, when the change in the image before and after the automatic operation has a difference determined by the automatic operation, it is determined that the automatic operation has been appropriately performed, and the difference in the image change before and after the automatic operation is determined by the automatic operation It may be determined that the automatic operation has not been appropriately performed when the difference is different.

In the erroneous input detection program according to the present invention,
In the change detection step, the change of the character string input to the automation target before and after the automatic operation is detected,
In the determination step, it is determined that the automatic operation has been appropriately performed when the change in the character string detected in the change detection step matches the character string input by the automatic operation device, and the character detected in the change detection step It may be determined that the automatic operation has not been appropriately performed when the change in the sequence does not match the character string input by the automatic operation device.

In the erroneous input detection program according to the present invention,
In the change detection step, the character string input by the automatic operation of the screen display to be automated is imaged, and the character string input by the automatic operation device is imaged,
In the determination step, when the character string image imaged in the change detection step matches, it is determined that the automatic operation has been appropriately performed, and the character string image imaged in the change detection step does not match It may be determined that the automatic operation has not been properly performed.

  According to the present invention, it is possible to suppress the risk of erroneous input and to reduce the time and labor required for confirmation work.

An example of a configuration of an automatic operation device according to the present embodiment is shown. 3 shows an example of a configuration of a result confirmation unit according to the first embodiment. The flowchart of the result confirmation method which concerns on Embodiment 1 is shown. 3 is an example of an image according to the first embodiment, where (a) shows a state before the mouse button is pressed down, (b) shows a state after the mouse button is pressed down, and (c) shows a state where the mouse button is pushed up. It shows the state after being done. The flowchart of the result confirmation method which concerns on Embodiment 2 is shown. FIG. 9 is an example of an image according to Embodiment 2, where (a) shows a state before the mouse button is pressed down, (b) shows a state after the mouse button is pressed down, and (c) shows a state where the mouse button is pushed up. It shows the state after being done. An example of the structure of the result confirmation part which concerns on Embodiment 3 is shown. The flowchart of the result confirmation method which concerns on Embodiment 3 is shown. It is an example of the image which concerns on Embodiment 3, (a) shows the state at the time of transmitting the character string A, (b) shows the state after inputting the character string A into the operation target window. An example of the structure of the result confirmation part which concerns on Embodiment 4 is shown. The flowchart of the result confirmation method which concerns on Embodiment 4 is shown. It is an example of the image which concerns on Embodiment 4, (a) shows the state at the time of transmitting the character string A, (b) shows the state after inputting the character string A into the operation target window. The flowchart of the result confirmation method which concerns on Embodiment 5 is shown. FIG. 10 is an example of an image according to the fifth embodiment, where (a) illustrates a state at the time when a character string A is transmitted, and (b) illustrates a state after the character string A is input to an operation target window. The flowchart of the result confirmation method which concerns on Embodiment 6 is shown. FIG. 10 is an example of an image according to Embodiment 6, where (a) shows a state at the time when a character string a is transmitted, and (b) shows a state after the character string A is input to an operation target window.

  Embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to the following embodiments. In the present specification and drawings, the same reference numerals denote the same components.

FIG. 1 shows an automatic operation device of the present invention. The automatic operation device 91 determines whether or not the automated operation has been correctly received by the automation target 92 based on the content displayed on the screen. The automation target 92 is an arbitrary device that automatically operates, for example, a computer. The automation target 92 may be an OS or an application. The contents of the determination can be exemplified by the following, for example.
・ Does the entered text match as a character?
・ Does the entered text match as an image?
・ Is the button pressed? (Did you dent and return?)
・ Is the check on / off changed?
-Has the screen changed?
• Did you hear a click sound?

  The automatic operation device 91 includes an operation detection unit 11, an automatic execution unit 12, and a result confirmation unit 13. The operation detection unit 11 acquires and records information on the operation performed by the user on the automation target 92 from the automation target 92. The automatic execution unit 12 automatically operates the automation target according to the recording. The result confirmation unit 13 functions as an erroneous input detection device and confirms that the automatic operation has been properly accepted by the automation target 92.

  For example, the result confirmation unit 13 includes a change detection unit and a determination unit. The change detection unit detects a change in the automation target 92 before and after the automatic operation on the automation target 92 performed by the automatic operation device 91. The determination unit determines whether the automatic operation has been appropriately performed based on the change detected by the change detection unit. The result confirmation unit 13 may execute the change detection step and the determination step by causing the computer to execute an erroneous input detection program, thereby realizing the change detection unit and the determination unit.

(Embodiment 1)
In the present embodiment, a case where the automatic execution unit 12 automatically clicks the mouse of the automation target 92 will be described. The result confirmation unit 13 determines whether or not the mouse click automatically executed on the automation target 92 is correctly recognized by the application of the automation target 92 based on a change in the image.

  In the case of a left click on the button-type GUI, it is confirmed that the image before the button down, after the button down, and after the button up changes appropriately. It is detected by comparing the images that the image near the click position changes after the button down and returns to the original after the button up, and the button is recognized.

  2 and 3 show a configuration diagram and a flow diagram of the result confirmation unit according to the present embodiment. The result confirmation unit 13 according to the present embodiment includes an image recognition unit 311 and an image storage unit 312 that function as a change detection unit, and an image comparison unit 313 that functions as a determination unit.

  The image recognition unit 311 recognizes the position of the screen display of the mouse included in the image shown in FIG. The screen display of the mouse is, for example, a display of an arrow having a specific shape in the screen display. The position is, for example, coordinates on the screen. When a plurality of windows are displayed on the screen, the position is within the window of the automation target 92. When the left mouse button is down (S101), the image recognition unit 311 captures the peripheral image (Y) of the mouse screen display (arrow) shown in FIG. 4B and stores it in the image storage unit 312 (see FIG. 4B). S102). When the left mouse button is raised (S103), the image recognition unit 311 captures the peripheral image (Z) of the mouse screen display (arrow) shown in FIG. 4C and stores it in the image storage unit 312 (see FIG. 4C). S104). The image comparison unit 313 compares the image (Y) and the image (Z) (S105). If the image (Y) and the image (Z) are different (Yes in S105), the image comparison unit 313 determines that the correct operation has been performed. If (Y) and image (Z) are equal (No in S105), it is determined that there is an erroneous input.

(Embodiment 2)
In the present embodiment, a case will be described in which the automatic execution unit 12 performs a left click on the GUI in the radio button format or the check box format. In this case, the result confirmation unit 13 confirms that the image before the button down and after the button up is appropriately changed. Specifically, attention is paid to a check mark indicating an ON / OFF state and a change in display / non-display of a black circle.

  2 and 5 show a configuration diagram and a flow diagram of the result confirmation unit according to the present embodiment. The image recognition unit 311 recognizes the position of the mouse screen display (arrow), captures a peripheral image (X) of the mouse screen display (arrow), as shown in FIG. (S201). When the left mouse button is clicked (up after down), the image recognition unit 311 captures a peripheral image (Z) of the mouse screen display (arrow) and stores it in the image storage unit 312 (S202 to S204). The image comparison unit 313 compares the image (X) and the image (Z) (S205). If the image (X) and the image (Z) are different (Yes in S205), the image comparison unit 313 determines that the correct operation has been performed. If (X) and image (Z) are equal (No in S205), it is determined that there is an erroneous input.

(Embodiment 3)
In the present embodiment, a case where the automatic execution unit 12 inputs a character string to the automation target 92 will be described. The result confirmation unit 13 compares the automatically input character string with the re-acquired character string that has been input and reflected on the screen, and confirms that the character string has been matched. judge.

  7 and 8 are a configuration diagram and a flow diagram of the result confirmation unit according to the present embodiment. The result confirmation unit 13 according to the present embodiment includes a character string acquisition unit 321 and a character string storage unit 322 that function as a change detection unit, and a character string comparison unit 323 that functions as a determination unit.

  The character string acquisition unit 321 displays the characters shown in FIG. 9B actually displayed on the screen after the automatic execution unit 12 transmits the character string (A1) shown in FIG. 9A to the automation target 92 (S301). The column (B1) is acquired and stored in the character string storage unit 322 (S302). The character string comparison unit 323 compares the character string (A1) held by the automatic execution unit 12 with the character string (B1) held by the character string storage unit 322 (S303). If the character string (A1) and the character string (B1) are different (No in S303), it is determined that an incorrect input has been made.

(Embodiment 4)
In the present embodiment, the automatic execution unit 12 inputs a character string to the automation target 92. The result confirmation unit 13 compares the image obtained by cutting out the input item portion from the hard copy of the screen on which the input character string is displayed with an image separately generated from the character string to be input, and matches or resembles the similarity. By confirming the above, it is determined that there are no missing characters or garbled characters.

  10 and 11 show a configuration diagram and a flow diagram of the result confirmation unit according to the present embodiment. The result confirmation unit 13 according to this embodiment includes an image recognition unit 331 that functions as a change detection unit, an image storage unit 332, an image generation unit 333, and an image comparison unit 334 that functions as a determination unit.

  After the automatic execution unit 12 transmits the character string (A1) shown in FIG. 12A (S401), the image recognition unit 331 displays the character string B1 shown in FIG. B2) is captured (S403), and the character string (A1) and the image (B2) are stored in the image storage unit 332. The image generation unit 333 generates an image (A2) from the character string (A1) held by the image storage unit 332 and stores it in the image storage unit 332 (S402). The image comparison unit 334 compares the image (A2) and the image (B2) (S404). If the image (A2) and the image (B2) are equal (Yes in S404), it is determined that the correct operation has been performed, and if the image (A2) and the image (B2) are different (No in S404), it is determined that the input is incorrect. .

  Note that the order of steps S402 and S403 is arbitrary. For example, S403 may be performed after step S402, or vice versa, or these may be performed in parallel.

(Embodiment 5)
In the present embodiment, the automatic execution unit 12 inputs a long character string to the automation target 92. When a series of character strings are displayed without being displayed at once, the result confirmation unit 13 makes a determination by comparing images and confirming a match or similarity each time a character is input. Specifically, an image of a single character that is input relative to the relative position is acquired based on the difference between the images before and after the input of one character, and a match with an image generated separately from the character is confirmed.

  10 and 13 are a configuration diagram and a flow diagram according to the present embodiment. After the automatic execution unit 12 transmits the character (a1) shown in FIG. 14A (S502), the image recognition unit 331 uses the character shown in FIG. 14B actually displayed on the screen as the image (b2). The image is captured and stored in the image storage unit 332 (S504). The display position of the image (b2) is also acquired and stored (S504). On the other hand, the image generation unit 333 generates an image (a2) from the character (a1) held by the image storage unit 332 and stores the image (a2) in the image storage unit 332 (S503). The image comparison unit 334 compares the image (a2) and the image (b2). If the image (a2) and the image (b2) are equal (Yes in S505), the image comparison unit 334 determines that the correct operation has been performed, and the image (a2) And the image (b2) are different (No in S505), it is determined as an erroneous input.

  Steps S502 to S505 are repeated for the number of characters included in the transmission character string (S501). At that time, in step S505, it is also confirmed from the display position of the image b2 that the arrangement of the characters a1 is appropriate. If the arrangement is inappropriate, it is determined that the input is incorrect.

  Note that the order of steps S503 and S504 is arbitrary. For example, S504 may be performed after step S503, or vice versa, or these may be performed in parallel.

(Embodiment 6)
When inputting a series of character strings, each time one character is input, it is easily determined that there is no character missing due to some change in the screen. Aim to improve processing load and processing speed at the expense of accuracy.

  10 and 15 show a configuration diagram and a flow diagram according to the present embodiment. The image recognition unit 331 captures the entire target screen shown in FIG. 16A before the automatic execution unit 12 transmits the character (a1), and stores it in the image storage unit 332 as an image (X) (S602). . Similarly, after the transmission of the character (a1), the entire target screen shown in FIG. 16B is captured and stored in the image storage unit 332 as an image (Y). The image comparison unit 334 compares the image (X) and the image (Y) (S605). If the image (X) and the image (Y) are different (Yes in S605), the image comparison unit 334 determines that the correct operation has been performed. When (X) and image (Y) are equal (No in S605), it is determined that there is an erroneous input. Steps S602 to S605 are repeated for the number of characters included in the transmission character string (S601).

(Embodiment 7)
In this embodiment, the registration part for setting operation | movement of the result confirmation part 13 is provided (not shown). The registration unit registers an arbitrary screen change as a determination condition of the image comparison unit. This makes it possible to determine whether other mouse and keyboard automatic operations have been correctly recognized by the target application.

(Embodiment 8)
In the present embodiment, the result confirmation unit 13 includes a confirmation sound generation unit (not shown). When the click sound is valid, the click sound is generated after the automatic execution unit 12 presses the button or the link. The confirmation sound generation unit recognizes this click sound and determines whether the automatic operation of the mouse and keyboard by the automatic execution unit 12 has been correctly recognized by the automation target 92.

(Embodiment 9)
In the present embodiment, the result confirmation unit 13 includes a condition selection unit (not shown) that enables selection of some of the conditions of the first to eighth embodiments. The condition selection unit determines whether or not all the selected conditions are satisfied, and determines that a correct operation is performed when all the conditions are satisfied, that is, when the AND condition is satisfied. Thereby, a stricter determination is possible.

(Embodiment 10)
In the present embodiment, the result confirmation unit 13 includes a condition selection unit (not shown) that enables selection of some of the conditions of the first to eighth embodiments. The condition selection unit determines whether any of the selected conditions is satisfied, and determines that a correct operation has been performed when any of the conditions is satisfied, that is, when the OR condition is satisfied. This enables more flexible determination.

  As described above, according to the present invention, since the result confirmation unit 13 collates changes before and after the automatic operation on behalf of a person, an extra confirmation assuming missing characters or garbled characters in automatic input of a character string by automation software. Work can be eliminated. There is no need to assume that the input of the mouse or keyboard operation is lost. In order to suppress these erroneous inputs, it is not necessary to take measures to slow down the processing speed, and the overall required time including the time required for the confirmation work can be shortened. The risk of incorrect information being registered in the system can be suppressed.

  The present invention can be applied to the information communication industry.

11: Operation detection unit 12: Automatic execution unit 13: Result confirmation unit 91: Automatic operation device 92: Automation target 311, 331: Image recognition unit 312, 332: Image storage unit 313, 334: Image comparison unit 321: Character string acquisition Unit 322: Character string storage unit 323: Character string comparison unit

Claims (8)

A change detection unit for detecting a change in the automation target before and after an automatic operation on the automation target performed by the automatic operation device;
A determination unit that determines whether or not an automatic operation has been appropriately performed based on the change detected by the change detection unit;
An erroneous input detection device comprising: The change detection unit images an operation part that performs an automatic operation in the screen display of the automation target, detects a change in the image before and after the automatic operation,
The determination unit determines that the automatic operation has been appropriately performed when the change in the image before and after the automatic operation has a difference determined by the automatic operation, and the difference in which the change in the image before and after the automatic operation is determined by the automatic operation The erroneous input detection device according to claim 1, wherein it is determined that the automatic operation has not been appropriately performed in a case where the operation is different. The change detection unit detects a change in a character string input to the automation target before and after an automatic operation,
The determination unit determines that the automatic operation is appropriately performed when the change in the character string detected by the change detection unit matches the character string input by the automatic operation device, and the character detected by the change detection unit. It is determined that the automatic operation has not been properly performed when the change in the column does not match the character string input by the automatic operation device.
The erroneous input detection device according to claim 1. The change detection unit images the character string input by the automatic operation in the screen display to be automated, and also images the character string input by the automatic operation device,
The determination unit determines that the automatic operation is appropriately performed when the image of the character string imaged by the change detection unit matches, and the image of the character string imaged by the change detection unit does not match The erroneous input detection device according to claim 1, wherein it is determined that the automatic operation has not been appropriately performed. A change detecting step for detecting a change in the automation target before and after an automatic operation on the automation target performed by the automatic operation device;
A determination step of determining whether or not automatic operation has been appropriately performed based on the change detected in the change detection step;
An erroneous input detection program for causing a computer to execute. In the change detection step, an operation part for performing an automatic operation in the screen display of the automation target is imaged, and a change in the image before and after the automatic operation is detected,
In the determination step, when the change in the image before and after the automatic operation has a difference determined by the automatic operation, it is determined that the automatic operation has been appropriately performed, and the difference in the image change before and after the automatic operation is determined by the automatic operation The erroneous input detection program according to claim 5, wherein it is determined that the automatic operation has not been appropriately performed when different from. In the change detection step, the change of the character string input to the automation target before and after the automatic operation is detected,
In the determination step, it is determined that the automatic operation has been appropriately performed when the change in the character string detected in the change detection step matches the character string input by the automatic operation device, and the character detected in the change detection step It is determined that the automatic operation has not been properly performed when the change in the column does not match the character string input by the automatic operation device.
The erroneous input detection program according to claim 5. In the change detection step, the character string input by the automatic operation of the screen display to be automated is imaged, and the character string input by the automatic operation device is imaged,
In the determination step, when the character string image imaged in the change detection step matches, it is determined that the automatic operation has been appropriately performed, and the character string image imaged in the change detection step does not match The erroneous input detection program according to claim 5, wherein it is determined that the automatic operation has not been appropriately performed.

Images