JP2019106093A - Computer, method of reproducing log, and storage medium - Google Patents

Abstract

To provide a computer, a method of reproducing a log, and a storage medium that securely record contents of an operation and perform verification when an operation record is reproduced.SOLUTION: A computer, which has a processor and a memory, includes: a log collection unit that hooks an operation on a screen output by a program operated by the computer and stores information on the operation as a log when the operation is a specific operation; and a log reproduction unit that reads the log and reproduces the operation on the screen when the program is in operation. The log collection unit, when the accepted operation is a specific operation, acquires a first capture image of a screen corresponding to the specific operation, extracts a first character string included in the first capture image, and stores the first character string in the log.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recording the operation of a computer and reproducing the recorded information.

  In computers, operations using a GUI (Graphical User Interface) in which an input device of a mouse and a keyboard and an image of a display device are used in common are in widespread use. Further, there is also known a technique for automating simple work by recording an operation on a GUI performed by a user and reproducing the recording (for example, Patent Documents 1 and 2).

  Patent Document 1 discloses a technique for verifying the result of automatic execution and the contents of the operation record depending on the environment in which the computer program is executed when the computer program is automatically executed based on the operation record.

  In patent document 1, when generating operation recording, a capture image is recorded triggered by the time of receiving the capture command of a screen. Then, at the time of reproducing the operation record, when the capture command is generated, the screen being reproduced is captured and compared with the captured image of the operation record to determine the presence or absence of abnormality.

  Further, in Patent Document 2, an event indicating a capture collection and storage instruction is registered in the operation record file, and when reproducing the operation record, the content of the screen being reproduced is captured and stored triggered by the event. Then, after reproducing the GUI operation by automatic execution, the user can confirm the display content by confirming the captured image, and can determine whether or not the GUI operation has been reproduced normally.

JP, 2013-143044, A Japanese Patent Application Publication No. 2007-25926

  In the above-described conventional example, the content of the operation is set in advance in the operation record (log) as a capture command or an event, so whether or not the operation is normally reproduced is determined for the operation without a capture command or an event. There was a problem that it was impossible.

  Accordingly, the present invention has been made in view of the above problems, and has an object of reliably recording the content of an operation and performing verification reliably at the time of reproduction of the operation recording.

  The present invention is a computer having a processor and a memory, and hooks an operation on a screen output by a program operated by the computer, and stores the information of the operation as a log when the operation is a specific operation. A log collecting unit, and a log reproducing unit that reads the log and reproduces an operation on the screen when the program is in operation, and the log collecting unit is configured to specify the received operation. In the case of operation, a first capture image of the screen corresponding to the specific operation is acquired, and a first character string included in the first capture image is extracted and stored in the log.

  Therefore, the present invention acquires captured images and character strings triggered by a specific operation (mouse operation or keyboard operation) when collecting logs, and captures images and character strings triggered by mouse operation or keyboard operations when reproducing logs. Get and compare. Thus, it is possible to determine that the operation on the computer has been reproduced properly, and the accuracy of verifying the operation can be improved.

It is a block diagram which shows the Example of this invention and shows an example of the computer which records operation reproduction and reproduction of operation. It is a flowchart which shows the Example of this invention and shows an example of the process performed with a computer. FIG. 6 shows an embodiment of the present invention and is a diagram showing an example of a log table. It is a flowchart which shows the Example of this invention and shows an example of the collection process of a log. It is a figure which shows the Example of this invention and shows an example of a log display screen. FIG. 8 is a diagram showing an example of the present invention and showing an example of a log display screen for setting a reproduction condition and a verification condition. It is a flowchart which shows the Example of this invention and shows an example of the reproduction process of a log. FIG. 6 shows an embodiment of the present invention and is a diagram showing an example of a verification condition. It is a figure which shows the Example of this invention and shows an example of a confirmation dialog. FIG. 8 is a diagram showing an example of the present invention and showing an example of a screen for setting a reproduction condition including stop. It is a flow chart which shows an example of the present invention and shows an example of processing in case a stop is included at the time of reproduction of a log. It is a figure which shows the Example of this invention and shows an example of a confirmation dialog. It is a flowchart which shows the Example of this invention and shows an example of a process in the case where repetition is included at the time of reproduction | regeneration of a log. FIG. 8 shows an embodiment of the present invention and is a diagram showing screen transition in the case where repetition is included at the time of reproduction of a log. FIG. 8 shows an embodiment of the present invention, and is a diagram showing screen transition in the case where repetition of changing a value when reproducing a log is included. FIG. 8 shows an embodiment of the present invention, and is a diagram showing screen transition in the case where repetition of changing input at the time of reproduction of a log is included.

  Hereinafter, embodiments of the present invention will be described based on the attached drawings.

  FIG. 1 shows an embodiment of the present invention, and is a block diagram showing an example of a computer that records a record of an operation on a computer as a log and reproduces the log at a desired timing.

  The computer 1 includes a CPU 11, a memory 12, an auxiliary storage device 13, a communication interface 14, an input interface 15 connected to an input device 21 including a keyboard and a mouse, and an output connected to a display device 20 such as a display. An interface 16 is included.

  The log collection unit 110, the image capture unit 111, the character string extraction unit 112, the log reproduction unit 113, and the application 114 are loaded as a program into the memory 12 and executed by the CPU 11. The auxiliary storage device 13 stores the collected log in the log table 120. The communication interface 14 is connected to a network (not shown).

  The log collection unit 110 acquires a log when the user 100 of the computer 1 operates the application 114 and records the log in the log table 120. The image capture unit 111 acquires an image output from the computer 1 to the display device 20 when acquiring a log, and stores the image in the auxiliary storage device 13. The log collecting unit 110 sends an instruction to acquire a captured image to the image capturing unit 111 at a predetermined timing. The capture image is information (for example, a bitmap or the like) in which the contents of the screen displayed at a specific time point are recorded as a snapshot.

  The character string extraction unit 112 extracts a character string from the image output by the log collection unit 110 and stores the character string in the auxiliary storage device 13. The log reproducing unit 113 reproduces the operation from the log of the log table 120 stored in the auxiliary storage device 13 with the application 114 or the OS (not shown) operated by the user 100.

  The CPU 11 operates as a functional unit that provides a predetermined function by performing processing according to the program of each functional unit. For example, the CPU 11 functions as a log collection unit 110 that processes according to the log collection program. The same is true for other programs. Furthermore, the CPU 11 also operates as a functional unit that provides each function of a plurality of processes executed by each program. A computer and a computer system are devices and systems including these functional units.

  FIG. 2 is a flowchart showing an example of a log collection process performed by the computer 1. This process can be started by the user 100 of the computer 1 at a desired timing. In addition, it is premised that the application 114 which acquires a log is started before starting this process. In addition, the operation of the user 100 with respect to the OS may be a target of the log.

  First, in step S1, the user 100 of the computer 1 activates the log collection unit 110 to start collecting logs of the application 114.

  In step S2, the log collecting unit 110 acquires the log (operation information) of the operation of the user 100 with respect to the application 114 and stores the acquired log in the log table 120 of the auxiliary storage device 13, as described later.

  In step S3, the user 100 of the computer 1 terminates the log collection unit 110 by a predetermined operation.

  In step S4, the log reproduction unit 113 reads the log table 120, generates a log display screen from the acquired log, and outputs the log display screen to the display device 20. In step S5, the user 100 sets log reproduction conditions or verification conditions on the log display screen. The log reproduction conditions include, for example, designation of a log to be reproduced, the number of repetitions for repeatedly executing reproduction, and change of an input operation to be reproduced. On the other hand, log verification conditions include information on string comparisons and value comparisons.

  In step S6, the log reproduction unit 113 executes the contents of the log of the log table 120 under the set reproduction condition, and reproduces the operation of the computer 1 performed by the user 100 on the application 114. Also, the log reproduction unit 113 verifies the extracted character string 217 of the log being reproduced based on the verification condition set on the log display screen. In step S7, the log reproduction unit 113 completes the reproduction of the range of the log specified by the user 100.

  In step S8, the log reproduction unit 113 inquires the user 100 whether or not to execute reproduction again. When execution of reproduction is requested again, the log reproduction unit 113 outputs the log display screen and then returns to step S5 to repeat the above processing. If the reproduction is to be ended, the process proceeds to step S9.

  In step S9, the log collection unit 110 inquires whether to collect a log of the operation. When collecting the log of operation, it returns to step S1 and repeats the above-mentioned processing. If not, the process ends.

  By the above processing, the log collection unit 110 accumulates the operation content for the application 114 operated by the user 100 of the computer 1 in the log table 120 as a log, and when the collection of the log is completed, the log reproduction unit 113 Load the log and execute the operation reproduction. When the log reproduction unit 113 executes the reproduction of the log, it is possible to specify the range of the log to be reproduced, the number of repetitions in the case where the reproduction is repeatedly performed, and change of the operation to be reproduced.

  FIG. 3 is a diagram showing an example of the log table 120 written by the log collection unit 110. As shown in FIG. The log table 120 includes time 121, target application 122, operation 123, target object 124, application position 125, application size 126, capture image path 127, enlarged image path 128, and extracted character string 129. Include in one entry.

  The time 121 stores the date and time when the log was acquired. The target application 122 stores the name (or identifier) of the application 114 that is active when the log is acquired.

  The operation 123 stores operation information acquired when a specific operation (mouse operation or keyboard operation) is received from the input device 21. When the operation of the input device 21 is a mouse operation, the coordinates (X, Y) on the screen of the display device 20 are stored. If the operation of the input device 21 is a keyboard operation, the received character code or character is stored. The target object 124 stores the names of elements to be operated, such as input fields.

  The application position 125 stores coordinates (X, Y) on the screen corresponding to the reference position (for example, the upper left base point) of the active application 114. The application size 126 stores the window size (width: W, height: H) on the screen of the active application 114.

  The captured image path 127 stores the path of the auxiliary storage device 13 in which the captured image is stored. The enlarged image path 128 stores the path of the auxiliary storage device 13 storing the enlarged captured image. The extracted character string 129 stores a character string extracted from the captured image or the enlarged captured image. The capture image and the enlarged capture image may be stored in the log table 120.

  FIG. 4 is a flowchart illustrating an example of the log collection process performed by the log collection unit 110. This process is a process performed in step S2 of FIG.

  The log collection unit 110 waits for the user 100 to operate the input device 21 (S1). When the input device 21 such as a mouse or a keyboard is operated, the process proceeds to step S11.

  In step S11, the log collection unit 110 hooks the operation of the input device 21 on the computer 1, and when the operation is a specific operation (mouse operation or keyboard operation), the operation information is acquired as a log. The operation of the input device 21 is an operation on a screen that the application 114 or the OS outputs to the display device 20. The technique for hooking the operation of the input device 21 may be a known technique or a well-known technique and thus will not be described in detail.

  When the mouse operation is a mouse click (specific operation), the log collection unit 110 acquires click coordinates (X, Y), and logs information such as the date and time of the application and the active application name in the log table 120. to add. The click coordinates (X, Y) are absolute coordinates of the screen of the display device 20.

  The log collecting unit 110 stores the date and time in the time 121 of the log table 120, stores the active application name in the target application 122, stores the click position in the operation 123, and transmits the name of the operation target to the target object 124. Then, the reference position of the application 114 is stored in the application position 125, and the window size of the application 114 is stored in the application size 126.

  In step S12, the image capture unit 111 captures the image of the active application 114, stores the captured image in the auxiliary storage device 13, and writes the storage destination in the captured image path 127 of the log table 120. The captured image may be an image of the entire screen of the display device 20 or an image of the active window of the application.

  In step S13, the character string extraction unit 112 extracts the character string included in the captured image acquired in step S12 and writes the extracted character string in the extracted character string 129 of the log table 120. In addition, the extraction of the character string from a capture image should just apply a well-known or well-known character recognition technique.

  In step S14, the image capture unit 111 captures the periphery of the click coordinates (X, Y) in a predetermined size, stores the captured image as the enlarged captured image in the auxiliary storage device 13, and stores the storage location in the enlarged image path 128 of the log table 120. Write to The enlarged capture image is, for example, a preset rectangular image including click coordinates, and is an image of an area such as 50 × 10 dots around the click coordinates.

  In step S15, the character string extraction unit 112 extracts the character string included in the enlarged captured image acquired in step S14 and adds the character string to the extracted character string 129 of the log table 120.

  In step S16, the log collection unit 110 hooks a keyboard operation (specific operation) to acquire operation information. The log collecting unit 110 adds information such as the value of the pressed key, the date and time, and the name of the active application to the log table 120.

  That is, the log collection unit 110 stores the date and time in the time 121 of the log table 120, stores the active application name in the target application 122, stores the value of the key in the operation 123, and targets the name of the operation target element. It stores in the object 124, stores the reference position of the application 114 in the application position 125, and stores the window size of the application 114 in the application size 126.

  In step S17, the image capture unit 111 captures an image of an application that is active at the time of keyboard operation, stores the captured image in the auxiliary storage device 13, and stores the storage location in the captured image path 127 of the log table 120. Write to The captured image is acquired in the same manner as the mouse operation.

  In step S18, the character string extraction unit 112 extracts the character string included in the captured image acquired in step S17 in the case of the keyboard operation, and writes the extracted character string in the extracted character string 129 of the log table 120.

  In step S19, the image capture unit 111 captures the periphery of the target object 124 for keyboard operation with a predetermined size, stores it as the enlarged captured image in the auxiliary storage device 13, and stores the storage destination in the enlarged image path 128 of the log table 120. Write. The enlarged captured image is similar to the enlarged captured image of the mouse operation.

  In step S20, in the case of keyboard operation, the character string extraction unit 112 extracts the character string included in the enlarged captured image acquired in step S19 and adds it to the extracted character string 129 of the log table 120.

  Next, in step S21, the log collection unit 110 determines whether to end collection of logs. If a predetermined keyboard operation or a predetermined mouse operation has not been performed, the process returns to step S10 to repeat the above processing to continue log collection. On the other hand, when a predetermined keyboard operation or a predetermined mouse operation is performed, the log collection process is ended.

  In the above process, the captured image 215 and the enlarged image 216 of the active application 114 are acquired in response to a mouse operation (mouse click) or a keyboard operation (key depression), and the character string included in each captured image is an extracted character string It is stored in 217.

  In the present embodiment, when a mouse operation or a keyboard operation occurs, the content of the operation is automatically accumulated in the log table 120 as a log.

  FIG. 5 is a diagram showing an example of the log display screen 210. As shown in FIG. This screen is output to the display device 20 in step S4 of FIG.

  The log display screen 210 includes the reproduction 211, the time 212, the target application 213, the operation 214, the capture image 215, the enlarged image 216, the extracted character string 217, the condition 218 and the stop 219 in one entry. Including.

  The reproduction 211 displays a check box for selecting a log to be reproduced when setting conditions for reproduction. The time 212 stores the time when the log is acquired, and corresponds to the time 121 of the log table 120.

  The target application 213 displays the name of the active application 114 at the time of log acquisition, and corresponds to the target application 122 in the log table 120. The operation 214 displays the coordinates of mouse click or the value of the keyboard operation key, and corresponds to the operation 123 of the log table 120.

  In the captured image 215, the captured image including the active application 114 is displayed in a reduced size. Captured image 215 is associated with the object at captured image path 127 of log table 120.

  In the magnified image 216, a magnified captured image of the application 114 to be subjected to mouse operation or keyboard operation is displayed in a reduced size. The magnified image 216 is associated with the object at the magnified image path 128 of the log table 120.

  The extraction string 217 displays a string included in the capture image or the enlarged capture image. The extracted character string 217 corresponds to the extracted character string 129 of the log table 120. In the condition 218, as described later, conditions for verifying a log to be reproduced can be set. In the stop 219, it is possible to specify a log to be paused when reproducing.

  The fields of the reproduction 211, the condition 218, and the stop 219 may be added to the log table 120, or may be stored in the memory 12 in association with the time 212.

  FIG. 6 is a view showing an example of the log display screen 210 when setting the reproduction condition and the verification condition. The log display screen 210 is output to the display device 20 in step S5 of FIG. When setting the reproduction condition, the user 100 operates the input device 21, puts a check in the check box of the reproduction 211 of the log display screen 210, and selects a line (log) for executing the reproduction of the operation. . In the present embodiment, an example is shown in which the lines L2 to L6 in the figure are selected as objects to be reproduced.

  Further, the user 100 can operate the input device 21 to set a verification condition in the condition 218 of the log display screen 210. In the illustrated example, it is set that the extracted character string 217 of the line L2 matches “application 1” as a verification condition at the time of reproduction, and the value of the extracted character string 217 of the line L6 is “111 ± 10%” Is set as a verification condition.

  In the present embodiment, the conditions of whether or not to reproduce each row, and conditions such as temporary suspension of reproduction are set as log reproduction conditions by the operation of the user 100, and the comparison of character strings at the time of reproduction is made. A condition for comparing values or values can be set as a verification condition.

  The log display screen 210 includes a repetition 302 for specifying the number of repetitions of reproduction and an execution button 301 for starting reproduction of the log. The user 100 can start the reproduction process by clicking the execution button 301 on the input device 21.

  FIG. 7 is a flowchart showing an example of log reproduction processing. This process is a process performed in step S6 of FIG.

  In step S31, the log reproduction unit 113 acquires information on the reproduction target row in which the check box of the reproduction 211 set on the log display screen 210 is checked and the verification condition set in the condition 218.

  In step S32, the log reproduction unit 113 sets the value of each field as a variable with reference to the log table 120 corresponding to the reproduction target row. That is, the log reproduction unit 113 sets the content of the target application 122 of the log table 120 to the target application Ap, sets the content of the operation 123 to the operation Op, sets the content of the target object 124 to the target object Ob, and the application The content of the position 125 is set to the application position Ps, the content of the application size 126 is set to the application size Sz, the content of the capture image path 127 is set to the capture image path P0, and the content of the enlarged image path 128 is an enlarged image path It is set to P1 and the content of the extracted character string 217 is set to the extracted character string Wd.

  Next, in step S33, the log reproduction unit 113 acquires a captured image from the auxiliary storage device 13 based on the captured image path P0, and acquires an enlarged captured image from the auxiliary storage device 13 based on the enlarged image path P1. Hold on 12.

  In step S34, the log reproduction unit 113 searches the target application Ap from the process currently being executed by the computer 1. Then, when the target application Ap is present during the execution process, the target application Ap is moved to the forefront (active), the position of the application 114 is set to the application position Ps, and the window size is set to the application size Sz. .

  On the other hand, when the target application Ap does not exist in the process currently being executed by the computer 1, the reproduction of the log is impossible, so the reproduction processing is ended.

  In step S35, if the content of the operation Op is coordinates, the log reproduction unit 113 moves the mouse pointer to the acquired coordinates because the operation is mouse operation. On the other hand, when the content of the operation Op is a keyboard operation, the log reproduction unit 113 sets the target object Ob as the input position.

  In step S36, the image capturing unit 111 captures an image of the active application 114 and sets it as a captured image C0. Next, the image capture unit 111 transmits the captured image C0 to the character string extraction unit 112. In step S37, the character string extraction unit 112 extracts the character string W0 from the received captured image C0.

  Next, in step S38, the image capturing unit 111 captures the coordinates of the mouse pointer or an image around the target object Ob at a predetermined size, and sets the captured image as a captured image C1.

  Next, the image capture unit 111 transmits the captured image C1 to the character string extraction unit 112. The character string extraction unit 112 extracts the character string W1 from the received captured image C1 in step S39.

In step S40, the log reproduction unit 113 compares the next object.
・ Capture image of capture image path P0 and capture image C0 acquired by reproduction processing
An enlarged captured image of enlarged image path P1 and a captured image C1 acquired by reproduction processing
・ Extraction character string Wd and character strings W0 and W1 acquired by reproduction processing

  If the comparison result of the three types of objects is a match or a partial match, the log reproduction unit 113 executes the mouse operation or the keyboard operation corresponding to the operation Op, and then proceeds to step S41. That is, the log reproducing unit 113 satisfies the verification condition of the captured image and the character string, determines that the reproduction of the log is normally performed, and reproduces the input of the log.

  On the other hand, when all the comparison results of the three types of objects do not match, the log reproduction unit 113 determines that the reproduction of the log is not properly performed, and outputs the dialog D1 shown in FIG. 9 to the display device 20. Do. When the user 100 selects "end" after outputting the dialog D1, the log reproduction unit 113 ends the reproduction processing. If the user 100 selects “continue”, the log reproduction unit 113 proceeds to step S41.

  In step S41, the log reproducing unit 113 determines whether or not the process has been performed for all the reproduction target lines, and if not completed, the process returns to step S32, and the above process is repeated for the next line. End the process.

  By the above process, it is possible to select a row selected from the log table 120 with the checkbox of the reproduction 211, and the log reproduction unit 113 acquires the captured image C0 acquired immediately before reproducing the operation Op and the enlarged captured image C1. These captured images are compared with the captured image (P0) acquired by the log collecting unit 110 and the enlarged captured image (P1), and the character string Wd acquired by the log collecting unit 110 and the log reproduction unit 113. The accuracy of the verification can be improved by comparing the acquired character strings W0 and W1.

  Thus, the log collecting unit 110 performs acquisition of a captured image and a character string triggered by a mouse operation or a keyboard operation, and the log reproducing unit 113 also reproduces the mouse operation or the keyboard operation according to the contents of the log table 120. Perform acquisition of the captured image and the character string before executing the operation Op.

  Then, by comparing the captured image and the character string acquired by the log collection unit 110 with the captured image and the character string acquired by the log reproduction unit 113, the content of the operation is recorded automatically and reliably, and the reproduction of the log is performed. Sometimes it is possible to do verifications reliably.

  Further, at the time of reproduction of the log, since the condition of verification can be set in the condition 218 on the log display screen 210 shown in FIG. 10, detection of an error or a problem can be easily realized. As the verification condition, contents as shown in FIG. 8 can be set.

  The verification conditions are defined as follows by the condition specification method 401 and the meaning 402 as shown in FIG. If the condition specification method 401 = "match", the log reproduction unit 113 satisfied the verification condition if the captured image 215 acquired in the log collection processing and the captured image C0 collected in the log reproduction processing match. It is determined that On the other hand, if the captured image 215 and the captured image C0 do not match, the log reproducing unit 113 outputs the dialog D1 shown in FIG. 9 to notify that the verification condition is not satisfied.

  In the case of the condition specification method 401 = "character string A, match", the log reproducing unit 113 satisfied the verification condition if the extracted character string 129 acquired in the log collection processing matches "character string A". It is determined that On the other hand, if the extracted character string 129 and the “character string A” do not match, the log reproducing unit 113 outputs the dialog D1 shown in FIG. 9 to notify that the verification condition is not satisfied.

  The verification condition in FIG. 7 shows an example in which “match” of the captured image and “character string A, match” are set as default verification conditions.

  In the case of the condition specification method 401 = "character string N0, character string N1, ..., match," multiple extracted character strings 129 acquired in the log collection process and "character string N0, character string N1, ..." If one of the character strings matches, the log reproducing unit 113 determines that the verification condition is satisfied. On the other hand, if the extracted character string 129 does not match any of “character string N0, character string N1,...”, The log reproducing unit 113 outputs the dialog D1 shown in FIG. 9 and the verification condition is not satisfied. Notify that.

  When the condition specification method 401 = "numeric value B ± 10", if the value obtained by converting the extracted character string 129 acquired in the log collection process into a numeric value is within the range of "numeric value B ± 10", the log reproduction unit 113 It is determined that the verification condition is satisfied. On the other hand, if the value obtained by converting the extracted character string 129 into a numerical value exceeds the range of “B ± 10”%, the log reproduction unit 113 outputs the dialog D1 shown in FIG. 9 and the verification condition is satisfied. Notify that you are not

  In the case of the condition specification method 401 = "more than C and less than D", the log reproduction unit 113 if the value obtained by converting the extracted character string 129 acquired in the log collection process into a value is more than C and less than D. It is determined that the verification condition is satisfied. On the other hand, if the value obtained by converting the extracted character string 129 into a numerical value exceeds the numerical value C or is smaller than the numerical value D, the log reproduction unit 113 outputs the dialog D1 shown in FIG. Notify that you are not

  In the case of the condition specification method 401 = “date E or more and date F or less”, the log reproduction unit 113 if the value obtained by converting the extracted character string 129 acquired in the log collection process to date is more than date E and less than date F. It is determined that the verification condition is satisfied. On the other hand, if the value obtained by converting the extracted character string 129 into a numerical value exceeds the date E or is less than the date F, the log reproduction unit 113 outputs the dialog D1 shown in FIG. 9 to satisfy the verification condition. Notify that you are not

<Stop process>
FIG. 10 is a view showing a modification 1 of the present embodiment, and showing an example of a log display screen 210 for setting a reproduction condition including a stop and a verification condition.

  The log display screen 210 is output to the display device 20 in step S5 of FIG. When setting the reproduction condition, the user 100 operates the input device 21 to put a check in the check box of the reproduction 211 of the log display screen 210 to select a line on which the reproduction of the operation is to be performed. In the illustrated example, lines (logs) of L1 to L4 in the figure indicate an example selected as a target of reproduction. The user 100 can start the reproduction process by clicking the execution button 301 on the input device 21.

  In addition, the user 100 can operate the input device 21 to check a check box for pausing the stop 219 of the log display screen 210. In the illustrated example, an example in which the reproduction of the log is temporarily stopped at line L3 is shown.

  By checking the check box at the stop 219 of the log display screen 210, the operation of the computer 1 can be verified by pausing the desired log.

  Further, the user 100 can operate the input device 21 to set the number of repetitions in the number of repetitions 302 of the log display screen 210. The user 100 sets a desired value to the number of repetitions 302 and clicks the execution button 301, whereby the log reproducing unit 113 repeats the reproduction to the number of repetitions 302 for the log for which the reproduction 211 is checked.

  FIG. 11 is a flowchart showing an example of a log reproduction process including stop. This process is obtained by replacing step S40 of the flowchart shown in FIG. 7 with step S50, and the other steps S31 to S39 and S41 are the same as those in FIG.

  In step S50, the log reproduction unit 113 compares the capture image of the capture image path P0 with the capture image C0 acquired by the reproduction processing, and compares the capture image of the enlarged image path P1 with the capture image C1 acquired by the reproduction processing Then, the extracted character string Wd is compared with the character strings W0 and W1 acquired in the reproduction processing. When the three comparison results indicate a match or a partial match, the log reproduction unit 113 determines whether the stop 219 (St in the drawing) of the log is checked.

  If the stop 219 (St in the figure) of the log is not checked, the log reproducing unit 113 executes the mouse operation or the keyboard operation in the same manner as in FIG. On the other hand, when the stop 219 (St in the drawing) of the log is checked, the log reproduction unit 113 displays the confirmation dialog D2 shown in FIG. 12 and waits for the operation of the user 100.

  After the user 100 executes the mouse operation or the keyboard operation by clicking “execute” in the confirmation dialog D2, the process proceeds to step S41. If the user 100 clicks "end" in the confirmation dialog D2, the process ends.

  If the three comparison results do not match, the log reproducing unit 113 displays the confirmation dialog D1 shown in FIG. 9 and waits for an instruction as to whether or not the reproduction is to be continued. If the received command is “continue”, the log reproducing unit 113 performs processing of mouse operation or keyboard operation, and then proceeds to step S41.

  As described above, the log reproduction unit 113 can perform the temporary stop with the log in which the stop 219 of the log display screen 210 is checked, and the user 100 temporarily stops the reproduction at a desired position to perform verification. be able to.

<Repetitive processing>
FIG. 13 is a flowchart illustrating an example of a log reproduction process including repetition. This process adds step S60 between step S40 and step S41 of the flowchart shown in FIG. 7, and the other steps S31 to S41 are the same as those in FIG.

  In step S60, when the value is set in the number of repetitions 302 of the log display screen 210 of FIG. 6 or 10, the log reproduction unit 113 reproduces the log whose reproduction 211 is checked up to the value of the number of repetitions 302. The process can be repeated.

  FIG. 14 is a diagram illustrating an example of log reproduction processing including repetition of the same operation. In the log display screen 210 in the figure, the reproduction 211 is checked for lines L1 to L3, and after the input of “100 + 10”, one character is deleted by “Backspace”, and then the processing of inputting “1” is repeated several times. An example of repeating "3 times" is shown. Note that a display 220 in the drawing indicates the value of the field to be operated.

  In the illustrated example, the log to be reproduced is repeated three times according to the contents of the log table 120, and from the log display screen 210-1 after the first repetition, the log display screen 210-2 after the second repetition, and 3 A transition is made to the log display screen 210-3 after the second repetition.

  FIG. 15 is a diagram showing an example of a log reproduction process in which a value is changed and repetition is performed. The log display screen 210 in the drawing shows an example in which the operation 214 of the line L3 changes the value from the variable c = 1 by the increment value c + = 2.

  First, on the log display screen 210-1 of the first repetition, after the input of “100 + 10”, one character is deleted by “Backspace”, and then “1” is input. In the log display screen 210-2 of the second repetition, after the input of "100 + 10", one character is deleted by "Backspace", and then "3" is input.

  Furthermore, on the third repeated log display screen 210-3, after the input of “100 + 10”, one character is deleted by “Backspace”, and then “5” is input.

  In this manner, after the contents of the log operation 214 are edited on the log display screen 210, it is possible to perform verification in the case where the pattern of reproduction of the log is changed by repeating and reproducing.

  FIG. 16 is a diagram showing an example of a log reproduction process in which the input is changed and repeated. The log display screen 210 in the figure shows an example in which the operation 214 of the line L3 changes the input by the value 1 and the increment value (3).

  First, on the log display screen 210-1 of the first repetition, after the input of “100 + 1”, one character is deleted by “Backspace”, and then “1” is input once. In the log display screen 210-2 of the second repetition, after the input of “100 + 1”, one character is deleted by “Backspace”, and then “1” is input three times.

  Furthermore, on the third repeated log display screen 210-3, after the input of “100 + 10”, one character is deleted by “Backspace”, and then “1” is input six times.

  As described above, after the contents of the operation 214 of the log are edited on the log display screen 210, the number of inputs is changed by repeatedly reproducing, and verification is performed when the pattern of reproduction of the log is changed. Can.

  Although the example in which the number of times of inputting “1” is changed is described above, the present invention is not limited to this. For example, the number of times and direction of cursor key input may be changed. Can be applied to the reproduction of

  As described above, according to the present embodiment, when a mouse operation or a keyboard operation occurs, the captured image 215 of the active application 114 and the enlarged image 216 are acquired, and then the character string included in each captured image is extracted It is stored in the column 217, and the content of the operation is automatically accumulated in the log table 120 as a log.

  Then, in the log reproduction process, the log reproduction unit 113 acquires, for the selected log, the captured image C0 acquired immediately before reproducing the operation Op and the enlarged captured image C1, and these captured images and the log The captured image (P0) acquired in the collection process is compared with the enlarged captured image (P1), and the character string Wd acquired in the log collection process and the character strings W0 and W1 acquired in the log reproduction process are further compared. The comparison makes it possible to determine that the operation on the computer 1 has been successfully reproduced, and the accuracy of verifying the operation can be improved.

  Also, by editing the operation 214, setting the number of repetitions 302 to execute the log reproduction processing can verify the operation on the computer 1 in the case where the value or the input is changed.

  That is, in the above-described conventional example, since the content of the operation record is only reproduced as it is, there is a problem that it is difficult to perform a test for changing the input operation using the operation record. On the other hand, in the present embodiment, by editing the operation 214 and performing repetitive reproduction processing, it is possible to perform reproduction processing that changes the operation or the value.

  For example, by accumulating work in the computer 1 in the log table 120 and performing log reproduction processing, it is possible to easily generate a procedure manual of the work. In addition, it is possible to extract problems in work using the computer 1, such as repetition of simple work.

  Also, if the log acquired by the computer 1 is read by another computer and the log reproducing unit 113 is operated by the other computer, the operation performed by the computer 1 can be performed by the other computer, for example, software It is possible to automate management tasks such as setting of

  In the above embodiment, an example in which the log of the operation is collected by the computer 1 and the log is reproduced by the computer 1 is shown, but one computer collects logs of a plurality of computers and the reproduction is performed by the computer. May be

  The present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are described in detail in order to illustrate the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. In addition, addition, deletion, or replacement of other configurations may be applied singly or in combination with some of the configurations of the respective embodiments.

  Further, each of the configurations, functions, processing units, processing means, and the like described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit. In addition, each configuration, function, and the like described above may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  Further, control lines and information lines indicate what is considered to be necessary for the description, and not all control lines and information lines in the product are necessarily shown. In practice, almost all configurations may be considered to be mutually connected.

1 Computer 11 CPU
12 Memory 13 Auxiliary Storage Device 20 Display Device 21 Input Device 110 Log Collection Unit 111 Image Capture Unit 112 Character String Extraction Unit 113 Log Reproduction Unit 114 Application 120 Log Table

Claims (15)

A computer having a processor and a memory,
A log collection unit that hooks an operation on a screen output by a program operated by the computer and stores information of the operation as a log when the operation is a specific operation;
A log reproducing unit that reads the log and reproduces an operation on the screen when the program is in operation;
The log collecting unit
A computer, comprising: acquiring a first capture image of a screen corresponding to the specific operation; extracting a first character string included in the first capture image; and storing the first character string in the log. The computer according to claim 1, wherein
The log reproduction unit
When the read log is a specific operation, a second capture image of the screen corresponding to the specific operation is acquired, and a second character string included in the second capture image is extracted, It is characterized in that the first captured image and the second captured image are compared, and the first character string and the second character string are compared to determine whether or not the operation has been successfully reproduced. calculator. The computer according to claim 1, wherein
The log reproduction unit
A computer characterized in that specification of a log for executing reproduction and a reproduction condition for executing reproduction are received, and the specified log is reproduced according to the reproduction condition. The computer according to claim 3, wherein
The reproduction condition is
A computer characterized in that it is the number of repetitions to repeat the reproduction. The computer according to claim 3, wherein
The reproduction condition is
A computer characterized by performing a pause during reproduction. The computer according to claim 4, wherein
The log reproduction unit
A computer characterized by editing an operation content of the log and changing a pattern of reproduction of the log. The computer according to claim 1, wherein
The log reproduction unit
A computer, receiving a verification condition to be compared with the first character string or the first capture image, and determining whether the log is reproduced properly based on the received verification condition. A method of reproducing a log in which a computer having a processor and memory reproduces the log,
A first step of hooking an operation on a screen output by a program operated by the computer;
The computer stores a second step of storing information of the operation as a log, when the hooked operation is a specific operation.
The computer includes a third step of reading the log and reproducing an operation on the screen when the program is in operation;
The second step is
A method of reproducing a log, comprising: acquiring a first capture image of a screen corresponding to the specific operation; extracting a first character string included in the first capture image; and storing the first character string in the log. . A method of reproducing a log according to claim 8, wherein
The third step is
When the read log is a specific operation, a second capture image of the screen corresponding to the specific operation is acquired, and a second character string included in the second capture image is extracted, It is characterized in that the first captured image and the second captured image are compared, and the first character string and the second character string are compared to determine whether or not the operation has been successfully reproduced. How to reproduce the log. A method of reproducing a log according to claim 8, wherein
The third step is
A log reproducing method comprising: specifying a log for executing reproduction; and receiving a reproduction condition for executing reproduction, and reproducing the designated log according to the reproduction condition. 11. A method of reproducing a log according to claim 10, wherein
The reproduction condition is
A method of reproducing a log characterized in that it is the number of repetitions of repetition of reproduction. 11. A method of reproducing a log according to claim 10, wherein
The reproduction condition is
A method of reproducing a log characterized in that a pause is performed during reproduction. A method of reproducing a log according to claim 11, wherein
The third step is
A method of reproducing a log, comprising editing an operation content of the log and changing a pattern of reproduction of the log. A method of reproducing a log according to claim 8, wherein
The third step is
Receiving a verification condition to be compared with the first character string or the first captured image, and based on the received verification condition, it is determined whether the log is reproduced normally or not. Method. A storage medium storing a program for reproducing a log by a computer having a processor and a memory,
A first step of hooking an operation on a screen output by a program operated by the computer;
If the hooked operation is a specific operation, a second step of storing information of the operation as a log;
And a third step of reading the log and reproducing an operation on the screen when the program is in operation;
The second step is
For acquiring the first capture image of the screen corresponding to the specific operation, extracting the first character string included in the first capture image and causing the computer to execute storing the first character string in the log Non-transitory computer readable storage medium storing a program.