JP2021111021A - Device test automation method, program and device - Google Patents

Abstract

To provide a technology for adding an input leakage operation to an operation record later.SOLUTION: A device test automation method includes a step for detecting an operation with respect to an operation screen of an objective device, and a step for determining a plurality of operation forms in the operation. The plurality of operation forms include a first operation form registered in an operation record, and an input-leakage second operation form not registered in the operation record. The method further includes; a step for recording first serial operations (1) to (3) including a screen transition which is performed in the first operation form; a step for accepting an operation including a screen transition performed in the second operation form; a step for accepting a start sign (X2) of an insertion operation; a step for accepting second serial operations (A) to (C) performed in the first operation form; a step for accepting a finish sign (X2) of the insertion operation; and a step for creating an operation record with the second serial operations (A) to (C) set as operations which have been accepted prior to the first serial operations (1) to (3).SELECTED DRAWING: Figure 9

Description

The present disclosure relates to a test of a device including an operation unit, and more specifically to a technique for generating an automatic execution script based on the operation content of the device.

In recent years, devices such as MFPs (Multifunction Peripheral) that accept input operations from users are becoming more multifunctional. Therefore, the man-hours for the operation test of these devices continue to increase. In particular, in software development, an increase in test man-hours is a heavy burden on manufacturers.

In the software development process, it is important to ensure the quality of software by repeating operation tests. However, the man-hours required for the operation test are finite, and in order to improve the development speed, the efficiency of the operation test is required. In order to improve the efficiency of the operation test, a test tool that records the operation contents and creates an automatic execution script that executes the same operation based on the operation contents may be used.

Regarding a test tool that automatically executes recorded operation contents, for example, Japanese Patent Application Laid-Open No. 2017-117309 (Patent Document 1) states that "application management of a management server transmits an application in which an operation event processing code is injected to a smartphone terminal. When a smartphone terminal operation is executed on the client terminal, the same operation is executed on the smartphone terminal, and the operation log information is saved in a script file by the script generation means. Test execution is performed on the client terminal. When instructed, the terminal management sends the application with the operation event processing code injected to multiple smartphone terminals to install it, and the script execution means reads the script from the script file, converts it into terminal operation information, and sends it to the smartphone terminal. The application is executed based on the received terminal operation information on the smartphone terminal that sends the application. ”(See [Summary]).

Further, Japanese Patent Application Laid-Open No. 2015-121869 (Patent Document 2) and Japanese Patent Application Laid-Open No. 2015-153183 (Patent Document 3) disclose techniques for operating history of operation contents.

JP-A-2017-117309 Japanese Unexamined Patent Publication No. 2015-121869 Japanese Unexamined Patent Publication No. 2015-153183

According to the techniques disclosed in Patent Documents 1 to 3, even if the tester notices that he / she has forgotten the operation to be performed in advance during the operation record, the forgotten operation can be added to the operation record later. Can not. As a result, the tester has to redo all the operations from the beginning, which increases the test man-hours. Therefore, when the tester notices that he / she has forgotten the operation to be performed in advance during the operation recording, a technique for adding the forgotten operation to the operation record later is required.

The present disclosure has been made in view of the above background, and the purpose in a certain aspect is forgotten when the tester notices in the operation record that he has forgotten the operation to be performed in advance. The purpose is to provide a technique for adding the operation that has been performed to the operation record later.

According to certain embodiments, methods for test automation of equipment are provided. This method includes a step of detecting an operation by the user on the operation screen of the target device that accepts the operation input of the user, and a step of determining a plurality of operation modes of the user in the operation. The plurality of operation modes include a first operation mode registered in the operation record and a second operation mode not registered in the operation record. The method includes a step of recording a first series of operations including a screen transition performed by the first operation mode, a step of accepting one or more operations including a screen transition performed by the second operation mode, and a step. The step of accepting the start signal of the insertion operation, the step of accepting the second series of operations according to the first operation mode, the step of accepting the end signal of the insertion operation, and the second series of operations are the first series of operations. It further includes a step of generating an operation record as being an operation received before.

In one aspect, the first series of operations involves a screen transition from the first screen to the second screen. One or more operations including the screen transition performed by the second operation mode include the operation of the screen transition from the second screen to the first screen. The step of accepting the start signal of the insertion operation is executed on the first screen. One or more operations including the screen transition executed by the second operation mode include a screen transition from the first screen to the third screen and a screen transition from the third screen to the first screen. .. The step of accepting the end signal of the insertion operation is executed on the first screen.

In a certain aspect, the step of generating the operation record receives a third series of operations including a screen transition from the first screen to the fourth screen before the first series of operations. It includes a step of generating an operation record assuming that the operations are received in the order of the third series of operations, the second series of operations, and the first series of operations.

In a certain aspect, the step of accepting the start signal of the insertion operation and the step of accepting the end signal of the insertion operation are the step of accepting the input by sound via the microphone, and the step of accepting the input by sound is the disclosure signal of the insertion operation. Alternatively, it includes a step of determining whether or not the input corresponds to the end signal.

In a certain aspect, the step of receiving the start signal of the insertion operation and the step of receiving the end signal of the insertion operation are the step of acquiring the movement of the user's hand via the camera and the step of receiving the movement of the user's hand, respectively. It includes a step of determining whether or not the movement corresponds to the disclosure signal or the end signal of the insertion operation.

In a certain aspect, the step of determining a plurality of operation modes is based on whether the finger touching the operation screen is either the finger of the right hand or the finger of the left hand, and each operation mode is the first operation mode or the first operation mode. A step of determining which of the two operation modes is included.

In a certain aspect, the step of determining a plurality of operation modes determines whether each operation mode is a first operation mode or a second operation mode based on the shape or color of the touch pen touching the operation screen. Includes steps to do.

In a certain aspect, the step of determining a plurality of operation modes is a step of determining whether each operation mode is a first operation mode or a second operation mode based on the shape of an object touching the operation screen. including.

According to other embodiments, another method for test automation of the device is provided. This method includes a step of detecting an operation by the user on the operation screen of the target device that accepts the operation input of the user, and a step of determining a plurality of operation modes of the user in the operation. The plurality of operation modes include a first operation mode registered in the operation record, and a second operation mode and a third operation mode not registered in the operation record. The method includes a step of recording a first series of operations including a screen transition performed by the first operation mode, a step of accepting one or more operations including a screen transition performed by the second operation mode, and a step. A step of accepting an insertion start operation according to the third operation mode, a step of accepting a second series of operations according to the first operation mode, a step of accepting an insertion end operation according to the third operation mode, and a second series of operations. It further includes a step of generating an operation record assuming that the operation is an operation received prior to the first series of operations.

In one aspect, the first series of operations involves a screen transition from the first screen to the second screen. One or more operations including the screen transition performed by the second operation mode include the operation of the screen transition from the second screen to the first screen. The insertion start operation is a screen transition from the first screen to the third screen, and one or more operations including the screen transition executed by the second operation mode are one or more operations on the third screen. Yes, the insertion end operation is a screen transition from the third screen to the first screen.

According to other embodiments, a program is provided that causes a computer to perform any of the above methods.

According to other embodiments, methods for test automation of equipment are provided. The device comprises a processor and a memory that stores a program that causes the processor to perform any of the methods described above.

According to an embodiment, if the tester notices during the operation record that he has forgotten an operation to be performed in advance, the forgotten operation can be added to the operation record later.

The above and other objectives, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

This is an example of an image forming device that generates an automatic execution script by a test script automatic generation system. It is a figure which shows an example of the test script automatic generation system according to a certain embodiment. It is a figure which shows an example of the hardware composition of the information processing apparatus 200. It is a figure which shows an example of the outline of the correction process of the evaluation operation based on the correction operation. It is a figure which shows the 1st example of the operation procedure of a user. It is a figure which shows an example of the internal operation at the time of detecting the operation of FIG. 5 in an information processing apparatus 200. It is a figure which shows a typical example of an operation error which cannot be corrected by a correction process. It is a figure for demonstrating an example of the internal processing of the information processing apparatus 200 necessary for correcting an operation error of FIG. 7. FIG. It is a figure which shows the 1st example of the insertion operation using the insertion start and insertion end signals. It is a figure which shows an example of the internal operation at the time of detecting the operation and the signal of FIG. 9 in the information processing apparatus 200. It is a figure which shows the 2nd example of the insertion operation using the insertion start and insertion end signals. It is a figure which shows an example of the internal operation at the time of detecting the operation and the signal of FIG. 11 in the information processing apparatus 200. It is a figure which shows an example of the generation procedure of the automatic execution script in an information processing apparatus 200. It is a figure which shows an example of the execution procedure of the operation information recording process in an information processing apparatus 200.

Hereinafter, embodiments of the technical concept according to the present disclosure will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

[About system configuration]
First, the outline of the test script automatic generation system according to the present embodiment will be described with reference to FIGS. In the following description, the processing contents of the test script automatic generation system will be described by taking an image forming apparatus as an example as a test object, but the test object of the test script automatic generation system according to the present embodiment is not limited to the image forming apparatus. The test script automatic generation system according to the present embodiment can be used for testing any device having an input means such as a touch panel.

FIG. 1 is an example of an image forming apparatus that generates an automatic execution script by a test script automatic generation system. The image forming apparatus 100 includes a control unit 101, a document reading unit 102, a print output unit 103, a fax (facsimile) unit 104, a storage device 105, a wireless interface 106, a wired interface 107, and a user authentication unit 108. And an operation panel 109. The control unit 101 includes a CPU (Central Processing Unit) 110, a RAM (Random Access Memory) 111, and a ROM (Read Only Memory) 112. The operation panel 109 includes a display unit 113 and an operation unit 114.

The control unit 101 controls the image forming apparatus 100 and executes various programs. The CPU 110 executes or refers to various programs and data read into the RAM 111. In a certain aspect, the CPU 110 may be an embedded CPU, an FPGA (Field-Programmable Gate Array), or a combination thereof. The CPU 110 can execute a program for realizing various functions of the image forming apparatus 100.

The RAM 111 stores a program executed by the control unit 101 and data referenced by the control unit 101. In a certain aspect, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) may be used as RAM 111.

The ROM 112 is a non-volatile memory and may store a program executed by the CPU 110. In that case, the CPU 110 executes the program read from the ROM 112 to the RAM 111. In certain aspects, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM) or a flash memory may be used as the ROM 112.

The document reading unit 102 can scan the document and convert the document into image data. The CPU 110 can acquire image data via the document reading unit 102. In a certain aspect, the document reading unit 102 may store the acquired image data in the storage device 105. In another aspect, the document reading unit 102 may acquire image data via the wired interface 107 and store the acquired image data in the storage device 105.

The print output unit 103 prints the image data captured by the image forming apparatus 100. In certain aspects, a circuit that controls various actuators for printing functions, including an imaging unit, may be used as the print output unit 103.

The FAX unit 104 transmits and receives manuscripts or image data by facsimile using a telephone line. In a certain aspect, a circuit having a facsimile communication control function and a communication port of a telephone line may be used as the FAX unit 104.

The storage device 105 is a non-volatile memory, and data can be stored even when the power of the image forming device 100 is turned off. The storage device 105 may store any program and data that the CPU 110 executes or references. In a certain aspect, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) may be used as the storage device 105. The CPU 110 can read various programs from the storage device 105 into the RAM 111 and execute the read programs as needed.

The wireless interface 106 is connected to a wired network device. In certain aspects, a Wi-Fi® module or the like may be used as the wireless interface 106. In another aspect, the wireless interface 106 may transmit and receive data using a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol) and UDP (User Datagram Protocol).

The wired interface 107 is connected to a wired network device. In some aspects, a wired LAN (Local Area Network) port or the like may be used as the wired interface 107. In another aspect, the wired interface 107 may send and receive data using communication protocols such as TCP / IP and UDP.

The user authentication unit 108 can authenticate the user based on the user ID (Identifier) and password input from the operation panel 109 or the like. In a certain aspect, the user authentication unit 108 may cooperate with an external authentication server, or may accept input of ecological authentication information such as a fingerprint instead of the user ID and password.

The operation panel 109 can accept various operations on the image forming apparatus 100 and display an operation menu and various information. As an example, the display unit 113 may be a liquid crystal monitor, an organic EL (Electro-Luminescence) monitor, or the like. The display unit 113 displays an operation menu and various information. As an example, the operation unit 114 is a touch sensor or a button incorporated in the operation panel 109, and can receive an operation input from the user.

FIG. 2 is a diagram showing an example of a test script automatic generation system according to the present embodiment. The test script automatic generation system according to the present embodiment records the user's operation in the image forming apparatus 100, and generates an automatic execution script for performing the same operation input to the image forming apparatus 100. The test script automatic generation system can accept correction of an erroneous operation when the user makes an erroneous operation while recording the user's operation in the image forming apparatus 100. In addition, the automatic test script generation system provides the ability to later insert a forgotten operation into the operation record. Hereinafter, along with the configuration of the test script automatic generation system, the correction process and the insertion process of the user's operation will be described.

The test script automatic generation system includes an information processing device 200, a camera 250, and a microphone 260. The camera 250 records or captures, for example, the screen of the operation panel 109 of the image forming apparatus 100 and the operation contents of the user. The information processing device 200 can generate an automatic execution script of the image forming device 100 based on the operation record 210 (video or photograph) acquired from the camera 250.

The microphone 260 is arranged near the screen of the operation panel 109 of the image forming apparatus 100, for example, and acquires a user's finger-leveling sound, a hand clapping sound, a voice, or the like. The information processing device 200 can execute an operation insertion process into an operation record, which will be described later, based on the detection of a specific sound by the microphone 260. In a certain aspect, instead of detecting a specific sound with the microphone 260, the information processing device 200 executes an operation insertion process into the operation record based on the user's specific hand movement taken by the camera 250. May be good. Further, in another aspect, the information processing apparatus 200 executes the process of inserting the operation into the operation record based on both the specific sound detected by the microphone 260 and the specific hand movement of the user captured by the camera 250. You may.

The information processing device 200 includes a screen analysis unit 201, an operation analysis unit 202, an operation determination unit 203, a correction processing unit 204, a script generation unit 205, a content DB (database) 206, a correction operation store 207, and the like. It includes an evaluation operation store 208, a script DB 209, and an insert operation store 215.

The screen analysis unit 201 analyzes what screen is included in the operation record 210 based on the operation record 210 acquired from the camera 250 and the content DB 206. As an example, the screen analysis unit 201 may determine that the screen included in the operation record 210 is a print setting screen, a scan setting screen, or the like.

The content DB 206 includes screen information on the operation screen of the image forming apparatus 100 and information on the arrangement of operation parts (input fields, pull-downs, buttons, etc.) included in each operation screen. The screen analysis unit 201 can compare the screen included in the operation record 210 with the operation screen included in the content DB 206, and detect on which screen the user is performing the operation input.

The operation analysis unit 202 can analyze the operation or signal performed by the user based on the operation record 210 acquired from the camera 250 and the content DB 206. The "operation" here refers to an act in which the user operates the touch panel or the like of the image forming apparatus 100 to cause the image forming apparatus 100 to perform some function. The "signal" here refers to an act of giving specific information to the information processing apparatus 200 and an act of not inputting to the image forming apparatus 100. In certain situations, actions such as "smoothing a finger," "clapping a hand," or "speaking a particular word" may be used as cues. The signal is used to make the information processing apparatus 200 recognize the start and end of the "insertion process" described later. Hereinafter, the operation mode from the start to the end of the "insertion process" of the information processing apparatus 200 is referred to as an "insertion mode", and the other operation modes are referred to as a "normal mode".

As an example, the screen analysis unit 201 analyzes the screen included in the operation record 210 and identifies the position touched by the user's finger. Next, the screen analysis unit 201 can detect what operation input the user is performing by referring to the content DB 206 and specifying the operation component arranged at the position touched by the user's finger.

The operation determination unit 203 can determine whether the user's operation is an "evaluation operation", a "correction operation", or a "signal" based on the user's operation mode. The operation mode is how the user inputs the operation. As an example, the operation modes are right-hand operation, left-hand operation, number of fingers, angle of finger as seen from camera 250, whether or not a touch pen is used, and an object (finger, touch pen, etc.) that operates the operation panel 109. ) And the like. As an example, the operation determination unit 203 determines whether the user's operation is an evaluation operation or a correction operation based on whether the finger touching the operation panel 109 is either the finger of the right hand or the finger of the left hand. Can be done. As another example, the operation determination unit 203 may determine whether the user's operation is an evaluation operation or a correction operation based on the shape or color of the touch pen that touches the operation panel 109. As yet another example, it may be determined whether the user's operation is an evaluation operation or a correction operation based on the shape of any object that touches the operation panel 109. Further, the operation determination unit 203 executes a function on the image forming apparatus 100 such as "smoothing a finger", "clapping a hand", or "sending a specific word" based on the information acquired from the camera 250 or the microphone 260. It is determined whether or not an operation mode that is not allowed to be detected is detected. The operation determination unit 203 can determine whether or not the user's operation is a "signal" based on the result of the determination.

The evaluation operation here is an operation that will be included in the automatic execution script. That is, the evaluation operation is an operation for operating the image forming apparatus 100. The correction operation is an operation for deleting an erroneous operation from the already input evaluation operations. The insert operation is an operation for inserting an operation that has been forgotten to be executed into an automatic execution script (or an operation record for generating an automatic execution script) later.

In a certain aspect, when the operation determination unit 203 determines that the evaluation operation has been performed when the user operates the image forming apparatus 100 with the right hand based on the operation record 210, and the user operates the image forming apparatus 100 with the left hand. It can be determined that the correction operation has been performed. In another aspect, the operation determination unit 203 determines the angle of the hand as seen from the camera 250, which finger used to operate the image forming apparatus 100, or how many fingers the image forming apparatus 100 operates based on the operation record 210. It is possible to determine whether the user's operation is an evaluation operation or a correction operation based on any method such as whether the operation is performed. Further, when the operation determination unit 203 detects the operation mode of the evaluation operation while the information processing apparatus 200 is operating in the insertion mode, the operation determination unit 203 determines that all of these operations are insertion operations.

In the following description, as an example, the operation determination unit 203 determines whether the user's operation is an evaluation operation or a correction operation based on which hand the user operates the image forming apparatus 100. It is explained as. Further, the operation determination unit 203 determines that the user's finger movement is a signal for switching between the "normal mode" and the "insertion mode".

The correction operation store 207 stores the correction operation temporary data that summarizes the correction operations. Correction operation temporary data is a temporary collection of correction operations within a series of operations for a certain operation test. When the operation determination unit 203 determines that the user has operated the image forming apparatus 100 with the left hand based on the operation record 210, the operation determination unit 203 can determine the operation input of the user as a correction operation. The correction operation is included in the correction operation temporary data in association with the operation component actually operated by the correction operation. The correction operation temporary data is stored in the correction operation store 207.

The evaluation operation store 208 stores evaluation operation temporary data that summarizes the evaluation operations. Evaluation operation temporary data is a temporary collection of evaluation operations within a series of operations for a certain operation test. When the operation determination unit 203 determines that the user has operated the image forming apparatus 100 with the right hand based on the operation record 210, the operation determination unit 203 stores the operation input of the user as an evaluation operation in the evaluation operation store 208. The evaluation operation is included in the evaluation operation temporary data in association with the operation component actually operated by the evaluation operation. The evaluation operation temporary data is stored in the evaluation operation store 208.

The insert operation store 215 collects the insert operations and stores the insert value operation temporary data. Insert operation temporary data is a temporary collection of insert operations within a series of operations for a certain operation test. When the operation determination unit 203 determines based on the operation record 210 that the user has given a signal to notify the start of the insertion process (when it is determined that the information processing apparatus 200 has entered the insertion mode), the operation determination unit 203 thereafter determines that the user has entered the insertion mode. The operation executed by the player according to the mode of the evaluation operation is stored in the insert operation store 215 as an insert operation. Insert operation Temporary data is used for the insert process described later.

The correction processing unit 204 corrects the evaluation operation temporary data based on the correction operation temporary data. More specifically, the correction processing unit 204 deletes some operations included in the evaluation operation temporary data based on the correction operation included in the correction operation temporary data.

Further, the correction processing unit 204 executes the “insertion processing” described later with reference to FIGS. 7 to 12. The "insertion process" is, for example, when the user notices that the operations (A) to (D) have been input to the image forming apparatus 100 and then the operations (E) to (F) have been forgotten to be input. , Refers to the process of inserting the operations (E) to (F) into the evaluation operation temporary data as the operations input before the operations (A) to (D).

The script generation unit 205 converts the modified evaluation operation temporary data into an automatic execution script that can be read into the image forming apparatus 100, and saves the automatic execution script in the script DB 209. In a certain aspect, the script generation unit 205 may output a list of incorrect operations by the user based on the correction operation temporary data. Further, in another aspect, the script generation unit 205 outputs information regarding each operation wrong by the user and the number of mistakes in each operation based on the past correction operation temporary data stored in the correction operation store 207. You may. The information can be used to improve the user interface of the image forming apparatus 100.

In some aspects, the test script automatic generation system may include configurations other than the camera 250 and the microphone 260 in order to determine the user's operation content. As an example, the test script automatic generation system may include distance measuring sensors on the upper right and upper left of the operation panel 109, and the finger position (operation position on the operation panel 109) may be specified from each of these distance sensors. Further, as another example, a touch panel sheet to be placed on the operation panel 109 may be provided, and the position of a finger (operation position on the operation panel 109) may be detected by the touch panel sheet. Further, the touch panel sheet may have a fingerprint detection function. The automatic test script generation system can distinguish between right-handed and left-handed operations by the fingerprint detection function. The test script automatic generation system may analyze the information acquired from these various sensors and the like as the operation record 210.

As described above, the test script automatic generation system according to the present embodiment can determine whether the user's operation is an evaluation operation, a correction operation, or a signal based on the user's operation mode. In addition, the test script auto-generation system can modify the evaluation operation based on the modification operation. In addition, the test script auto-generation system can insert into the evaluation operation an operation that the user has forgotten to perform later, based on the insert operation. Therefore, even if the user makes an erroneous operation during the test operation, it is not necessary to restart the operation from the beginning, and the operation can be corrected on the spot. As a result, the test script automatic generation system can greatly improve the efficiency of creating the automatic execution script in the image forming apparatus 100 having many functions.

Each functional unit of the information processing apparatus 200 of FIG. 2 may be realized as a program or data executed by the hardware described with reference to FIG. FIG. 3 is a diagram showing an example of the hardware configuration of the information processing apparatus 200.

The information processing device 200 includes a CPU 301, a primary storage device 302, a secondary storage device 303, an external device interface 304, an input interface 305, an output interface 306, and a communication interface 307.

The CPU 301 executes or refers to various programs and data read into the primary storage device 302. In a certain aspect, the CPU 301 may be an embedded CPU, an FPGA, or a combination thereof. The CPU 301 can execute a program for realizing the various functional units shown in FIG.

The primary storage device 302 stores a program executed by the CPU 301 and data referenced by the CPU 301. In certain aspects, DRAM or SRAM may be used as the primary storage device 302.

The secondary storage device 303 is a non-volatile memory, and can store data even when the power of the information processing device 200 is turned off. The secondary storage device 303 may store any program and data executed or referenced by the CPU 301. In some aspects, the HDD or SSD may be used as the secondary storage device 303. The CPU 301 can read various programs from the secondary storage device 303 into the primary storage device 302 as needed, and execute the read programs.

The external device interface 304 may be connected to any external device such as a printer, scanner and external HDD. The information processing device 200 may communicate with the camera 250 and the microphone 260 via the external device interface 304. In a certain aspect, a USB (Universal Serial Bus) terminal or the like may be used as the external device interface 304.

The input interface 305 may be connected to any input device such as a keyboard, mouse or gamepad. In some aspects, a USB terminal, a PS / 2 terminal, a Bluetooth® module, and the like may be used as the input interface 305.

The output interface 306 may be connected to any output device such as a cathode ray tube display, a liquid crystal display or an organic EL display. In some aspects, a USB terminal, a D-sub terminal, a DVI terminal, an HDMI® terminal, and the like may be used as the output interface 306.

The communication interface 307 is connected to a wired or wireless network device. The information processing device 200 may communicate with the camera 250 and the microphone 260 via the communication interface 307. In some aspects, a wired LAN (Local Area Network) port, a Wi-Fi module, or the like may be used as the communication interface 307. In another aspect, the communication interface 307 may send and receive data using a communication protocol such as TCP / IP or UDP.

[Correction processing of evaluation operation]
FIG. 4 is a diagram showing an example of an outline of the correction process of the evaluation operation based on the correction operation. With reference to FIG. 4, the roles of the evaluation operation and the correction operation of the test script automatic generation system according to the present embodiment will be described.

On the screen 401, first, the user performs the operation (A) and the operation (B) with the right hand (evaluation operation). The information processing apparatus 200 includes operations (A) and B in the evaluation operation temporary data based on the operation record acquired from the camera 250. Second, the user notices that the operation (B) is an incorrect operation, and cancels the operation (B) with his / her left hand (correction operation). The information processing apparatus 200 includes the correction operation of the operation (B) in the correction operation temporary data based on the operation record acquired from the camera 250. Third, the information processing apparatus 200 deletes the operation (B) from the evaluation operation temporary data based on the correction operation temporary data, and generates an automatic execution script from the modified evaluation operation temporary data. As mentioned above, evaluation operations are used to record the operations that should be included in the autorun script. The corrective operation can be used to overwrite the wrong evaluation operation with the correct evaluation operation.

Next, with reference to FIGS. 5 and 6, a typical example of an operation error that can be corrected by the correction process of the evaluation operation will be described. The operation that can be corrected by the correction process of the evaluation operation is typically an operation error immediately before.

FIG. 5 is a diagram showing a first example of a user operation procedure. In the example of FIG. 5, the user makes an error once in the operation of all nine steps, and immediately after that, performs a correction operation.

On the home menu screen 501, the user performs the evaluation operation (1) "copy selection" with the right hand. The screen transitions to the copy setting screen 502 by the evaluation operation (1). On the copy setting screen 502, the user performs the evaluation operation (2) "set the number of copies to 3" and the evaluation operation (3) "set the color setting to color" with the right hand.

Immediately after the evaluation operation (3), the user notices that the color setting was originally planned to be set to monochrome. Therefore, the user performs the correction operation (A) "Cancel color setting" with the left hand. Next, with the right hand, perform the evaluation operation (3') "Set the color setting to monochrome".

Subsequently, on the copy setting screen 502, the user performs the evaluation operation (4) "set the print type on both sides" and the evaluation operation (5) "selection of the margin setting button" with the right hand. By the evaluation operation (5), the screen transitions to the margin setting screen 503.

On the margin setting screen 503, the user uses his right hand to perform the evaluation operation (6) "set the vertical margin to 5 mm" and the evaluation operation (7) "set the horizontal margin to 5 mm" and the evaluation operation (8). Perform "OK selection". The screen transitions to the copy setting screen 502 by the evaluation operation (8). On the copy setting screen 502, the user performs the evaluation operation (9) “print execution” with his / her right hand.

FIG. 6 is a diagram showing an example of an internal operation when the operation of FIG. 5 in the information processing apparatus 200 is detected. The operation record 601 includes evaluation operations (1) to (9) and (3'), and a correction operation (A). The internal processing of the information processing apparatus 200 based on the operation of FIG. 5 will be described below.

In the first step, the information processing apparatus 200 generates the evaluation operation temporary data 602 and the correction operation temporary data 603 based on the operation record 601 acquired from the camera 250. The evaluation operation temporary data 602 includes the operation (3) and the operation (3') for the "color setting" as a record. The correction operation temporary data 603 includes the operation (A) for correcting the "color setting" as a record.

In the second step, the information processing apparatus 200 modifies the evaluation operation temporary data 602 based on the modification operation temporary data 603. More specifically, first, the information processing apparatus 200 performs the "color setting" immediately before the correction operation (A) in the evaluation operation temporary data 602 based on the correction operation (A) for the "color setting". The evaluation operation (3) for is deleted. Next, the information processing apparatus 200 is based on the evaluation operation (3') for the "color setting" performed after the correction operation (A), and the position where the evaluation operation (3) is in the evaluation operation temporary data 602. Insert the evaluation operation (3') into. By this process, the information processing apparatus 200 finally replaces the evaluation operation (3) of the evaluation operation temporary data 602 with the evaluation operation (3').

In the third step, the information processing apparatus 200 converts the modified evaluation operation temporary data 602 into an automatic execution script. The information processing device 200 can store the generated automatic execution script in the secondary storage device 303. The user can reproduce the operation based on the operation of FIG. 5 in the image forming apparatus 100 by inputting the automatic execution script into the image forming apparatus 100 via the network, for example.

As described above, the information processing apparatus 200 performs the evaluation operation (X) on the operation component (P) and then detects the correction operation (Y) on the operation component (P), and then the evaluation operation (X). ) Is deleted. Alternatively, when the operation component (P) is an operation component for setting selection, the information processing apparatus 200 sets the operation component (P) based on the detection of the correction operation (Y) for the operation component (P). To return to the default.

Further, the information processing apparatus 200 detects the correction operation (Y) for the operation component (P) and then detects the evaluation operation (Z) for the operation component (P) again, and based on that, the evaluation operation temporary data The evaluation operation (Z) is inserted at the position where the evaluation operation (X) was performed. That is, the information processing apparatus 200 overwrites the evaluation operation (X) in the evaluation operation temporary data with the evaluation operation (Z).

By these processes, even if the user makes a mistake in the middle of the operation, it is not necessary to restart the operation from 1, and the target automatic execution script can be obtained only by performing the correction operation and the overwrite operation.

The conventional automatic test script generation system does not have a function of modifying the evaluation operation based on such an operation mode. Therefore, the user cannot correct the evaluation operation in the middle, and if the user makes a mistake in inputting the evaluation operation even once, he / she has to start over from the beginning.

For example, if the user uses a conventional test script auto-generation system to create an auto-execution script that includes an operation of all 30 steps, if the user fails the operation at the 29th step, the user redoes all 30 steps. I needed it.

On the other hand, when the user creates an automatic execution script including the operation of all 30 steps using the test script automatic generation system according to the present embodiment, even if the user fails the operation at the 29th step, it is corrected. The operation only needs to be executed once, and the burden of creating the automatic execution script of the user can be significantly reduced.

[About the insertion process of evaluation operation]
Next, an example of the insertion process will be described with reference to FIGS. 7 to 12. The information processing apparatus 200 can correct an operation error by executing an insertion process when an operation error that cannot be corrected by the correction process occurs.

FIG. 7 is a diagram showing a typical example of an operation error that cannot be corrected by the correction process. In the example of FIG. 7, the user notices that the evaluation operations (A) to (C) should have been performed in advance during the evaluation operations (1) to (5).

On the home menu screen 701, the user performs the evaluation operation (1) "scan selection" with the right hand. The screen transitions to the scan setting screen 702 by the evaluation operation (1). On the scan setting screen 702, the user performs the evaluation operation (2) "set the resolution setting to the standard" and the evaluation operation (3) "set the color setting to the color" with the right hand.

The user was supposed to perform the evaluation operation (4) "selection of the scan data transmission destination" after the evaluation operation (3), but forgot to register the scan data transmission destination in the image forming apparatus 100 in advance. Notice that it was.

Originally, the user needs to perform the evaluation operation (A) "setting selection" on the home menu screen 701 before the evaluation operation (1). The screen transitions to the initial setting screen 703 by the evaluation operation (A). On the initial setting screen 703, the user performs the evaluation operation (B) "registering the e-mail address" with the right hand, and registers the e-mail address to be selected in the evaluation operation (4) in the image forming apparatus 100. Next, the screen is changed to the home menu screen 701 by performing the evaluation operation (C) "selection of OK button". The user can input the correct setting to the image forming apparatus 100 by executing the evaluation operations (1) to (5) after executing these evaluation operations (A) to (C).

As described above, when the user forgets to input the operation that should be performed and performs the next operation, the operation error cannot be corrected even if the correction operation described with reference to FIGS. 5 to 6 is executed. .. This is because the correction operation is an operation for correcting the content of the evaluation operation that is input incorrectly, and the evaluation operation that is not input later cannot be added to the evaluation operation temporary data. That is, the correction operation can overwrite the contents of the evaluation operations (1) to (5) input to the image forming apparatus 100, but the evaluation operations (A) to (A) to (5) not input to the image forming apparatus 100. B) cannot be inserted into the evaluation operation temporary data as an operation input before the evaluation operations (1) to (5).

FIG. 8 is a diagram for explaining an example of internal processing of the information processing apparatus 200 necessary for correcting the operation error of FIG. 7. The information processing device 200 evaluates a series of operations 810 for registering the e-mail address input after the evaluation operations (1) to (5). A series of operations 820 for setting the scan in the evaluation operation temporary data 802. Must be inserted before. A signal described later may be used for the insertion process.

FIG. 9 is a diagram showing a first example of an insertion operation using the insertion start and insertion end cues. In the example of FIG. 9, the user noticed that the evaluation operations (A) to (C) should have been performed in advance during the operations of the evaluation operations (1) to (5), and the evaluation operation temporary data. The evaluation operations (A) to (C) are inserted immediately before the evaluation operation (1) in the above.

First, on the home menu screen 701, the user performs the evaluation operation (1) "scan selection" with the right hand. The screen transitions to the scan setting screen 702 by the evaluation operation (1). On the scan setting screen 702, the user performs the evaluation operation (2) "set the resolution setting to the standard" and the evaluation operation (3) "set the color setting to the color" with the right hand.

Immediately after the evaluation operation (3), the user notices that he / she has forgotten to register the transmission destination of the scan data in the image forming apparatus 100 in advance. Therefore, the user shifts the screen to the home menu screen 701 by performing the correction operation (4) "selection of the cancel button" with the left hand. The correction operation (4) is not recorded in the evaluation operation temporary data.

Next, the user smoothes his finger to execute the insertion start signal (X1). The information processing device 200 detects the insertion start signal (X1) by the camera 250 or the microphone 260, and recognizes that the operations input to the image forming device 100 after that are the operations to be inserted. More specifically, the information processing apparatus 200 determines whether or not the sound input from the microphone 260 is a sound corresponding to the insertion start signal (X1). In a certain aspect, the information processing apparatus 200 may determine whether or not the user's action corresponding to the insertion start signal (X1) can be detected from the video or image acquired from the camera 250.

Next, on the home menu screen 701, the user performs the evaluation operation (A) "setting selection" with the right hand. The screen transitions to the initial setting screen 703 by the evaluation operation (A). On the initial setting screen 703, the user performs the evaluation operation (B) "registering the e-mail address" with the right hand, and registers the e-mail address in the image forming apparatus 100. Next, the user shifts the screen to the home menu screen 701 by performing the evaluation operation (C) "selection of the OK button" with the right hand.

Next, the user smoothes his / her finger to execute the insertion end signal (X2). The information processing device 200 detects the insertion end signal (X2) by the camera 250 or the microphone 260, and recognizes that the operation input to the image forming device 100 after that is a normal evaluation operation. In some aspects, the insertion start signal (X1) and the insertion end signal (X2) may be clapping hands or uttering specific words. More specifically, the information processing apparatus 200 determines whether or not the sound input from the microphone 260 is a sound corresponding to the insertion end signal (X2). In a certain aspect, the information processing apparatus 200 may determine whether or not the user's action corresponding to the insertion end signal (X2) can be detected from the video or image acquired from the camera 250.

As described above, the information processing apparatus 200 detects two cues in addition to the evaluation operation and the correction operation, so that the operation input to the image forming apparatus 100 between the first cues and the second cues is performed. Can be determined to be an operation that is the target of the insertion process. That is, the information processing device 200 has been able to detect the series of operations 810 of FIG. 8 by detecting the two signals.

Next, on the home menu screen 701, the user performs the correction operation (5) "scan selection" with the left hand. By the correction operation (5), the screen transitions to the scan setting screen 702. The correction operation (5) is not recorded in the evaluation operation temporary data. Next, on the scan setting screen 702, the user performs the evaluation operation (6) "selection of the scan data transmission destination" and the evaluation operation (7) "selection of the OK button" with the right hand.

FIG. 10 is a diagram showing an example of an internal operation when the operation and the signal of FIG. 9 in the information processing apparatus 200 are detected. In the first step, the information processing apparatus 200 detects the evaluation operations (1) to (3). The information processing apparatus 200 records a series of operations 1001 including the evaluation operations (1) to (3) in the evaluation operation temporary data 1006.

In the second step, the information processing apparatus 200 detects the correction operation (4) and the insertion start signal (X1). Since the correction operation (4) and the insertion start signal (X1) are not the operations to be evaluated, the information processing apparatus 200 does not record the correction operation (4) and the insertion start signal (X1) in the evaluation operation temporary data 1006. Further, the information processing apparatus 200 recognizes that the subsequent evaluation operations are operations that are subject to the insertion process.

In the third step, the information processing apparatus 200 detects the evaluation operations (A) to (C). Since the evaluation operations (A) to (C) are operations after the insertion start signal (X1), the information processing apparatus 200 targets a series of operations 1003 including the evaluation operations (A) to (C) for the insertion process. A series of operations 1003 including the evaluation operations (A) to (C) are recorded in the insertion operation temporary data.

In the fourth step, the information processing apparatus 200 detects the insertion end signal (X2) and the correction operation (5). Since the insertion end signal (X2) and the correction operation (5) are not the operations to be evaluated, the information processing apparatus 200 does not record the insertion end signal (X2) and the correction operation (5) in the evaluation operation temporary data 1006. Further, the information processing apparatus 200 recognizes that the subsequent evaluation operations are normal evaluation operations.

In the fifth step, since the range of the series of operations 1003 (insertion operation temporary data) to be the target of the insertion process is determined, the information processing apparatus 200 inserts the series of operations 1003 at the beginning of the evaluation operation temporary data.

In the sixth step, the evaluation operations (6) and (7) are detected. The information processing apparatus 200 adds a series of operations 1005 including the evaluation operations (6) and (7) to the evaluation operation temporary data 1006.

As described above, the information processing apparatus 200 recognizes a series of operations 1003 to be the target of the insertion process by the insertion start signal (X1) and the insertion end signal (X2). Then, the information processing apparatus 200 inserts a series of operations 1003, which is the target of the insertion process, at the beginning of the evaluation operation temporary data 1006. By this process, the user can later insert the initial setting operation into the evaluation operation temporary data when the initial setting is forgotten during the operation of the test target device (image forming apparatus 100). As a result, even if there is an input omission in the test operation, the user does not have to restart the operation from the beginning, and the test man-hours of the device to be tested can be reduced.

In the example of FIG. 10, the information processing apparatus 200 inserts a series of operations 1003 to be the target of the insertion process at the beginning of the evaluation operation temporary data 1006, but this is just an example and a series of operations to be the target of the insertion process. The insertion position of 1003 is not limited to this. In a certain aspect, the information processing apparatus 200 may determine the insertion position of a series of operations 1003 to be inserted, based on the transition information of the screen. Details of the determination of the insertion position will be described with reference to FIG.

Further, the information processing apparatus 200 detects the insertion start signal (X1) and the insertion end signal (X2) in order to identify the series of operations 1003 to be the target of the insertion process, but is the target of the insert process. The identification method of the series of operations 1003 is not limited to this. In a certain aspect, the information processing apparatus 200 may detect a third operation mode different from both the evaluation operation and the correction operation as the insertion start operation (Y1) and the insertion end operation (Y2). For example, the information processing device 200 may recognize the operation with two fingers as the third operation mode. In that case, the information processing apparatus 200 may detect the insertion start operation (Y1) "selection of the cancel button" with two fingers of the right hand instead of the correction operation (4) and the insertion start signal (X1). Similarly, the information processing apparatus 200 may detect the insertion end operation (Y2) "scan selection" with two fingers of the right hand instead of the insertion end signal (X1) and the correction operation (5). The information processing apparatus 200 can reduce the number of user inputs during the insertion process by detecting the insertion start operation (Y1) and the insertion end operation (Y2) in any operation mode. Further, in another aspect, the insertion start operation (Y1) and the insertion end operation (Y2) may be different operation modes from each other. The same applies to the processing after FIG. 11.

FIG. 11 is a diagram showing a second example of an insertion operation using the insertion start and insertion end cues. In the example of FIG. 11, when the user completes the evaluation operations (1) to (3) and then executes the evaluation operations (4) to (6), the evaluation operations (A) to (C) are performed in advance. I realize that I need to do it in advance.

On the home menu screen 1101, the user performs the evaluation operation (1) "copy selection" with the right hand. The screen transitions to the copy setting screen 1102 by the evaluation operation (1). On the copy setting screen 1102, the user performs the evaluation operation (2) "set the number of copies to 2" and the evaluation operation (3) "selection of the OK button" with the right hand. The screen transitions to the home menu screen 1101 by the evaluation operation (3).

Subsequently, on the home menu screen 1101, the user performs the evaluation operation (4) "scan selection" with the right hand. By the evaluation operation (4), the screen transitions to the scan setting screen 1103. On the scan setting screen 1103, the user performs the evaluation operation (5) "set the resolution setting to the standard" and the evaluation operation (6) "set the color setting to the color" with the right hand.

Immediately after the evaluation operation (6), the user notices that he / she has forgotten to register the transmission destination of the scan data in the image forming apparatus 100 in advance. Therefore, the user shifts the screen to the home menu screen 1101 by performing the correction operation (7) "selection of the cancel button" with the left hand. The correction operation (7) is not recorded in the evaluation operation temporary data.

Next, the user smoothes his finger to execute the insertion start signal (X1). The information processing device 200 detects the insertion start signal (X1) by the camera 250 or the microphone 260, and recognizes that the operations input to the image forming device 100 after that are the operations to be inserted.

Next, on the home menu screen 1101, the user performs the evaluation operation (A) "setting selection" with the right hand. The screen transitions to the initial setting screen 1104 by the evaluation operation (A). On the initial setting screen 1104, the user performs the evaluation operation (B) "registering the e-mail address" with the right hand, and registers the e-mail address in the image forming apparatus 100. Next, the user shifts the screen to the home menu screen 1101 by performing the evaluation operation (C) "selection of the OK button" with the right hand.

Next, the user smoothes his / her finger to execute the insertion end signal (X2). The information processing device 200 detects the insertion end signal (X2) by the camera 250 or the microphone 260, and recognizes that the operation input to the image forming device 100 after that is a normal evaluation operation.

Next, on the home menu screen 1101, the user performs the correction operation (8) "scan selection" with the left hand. The screen transitions to the scan setting screen 1103 by the correction operation (8). The correction operation (8) is not recorded in the evaluation operation temporary data. Next, on the scan setting screen 1103, the user performs the evaluation operation (9) "selection of the scan data transmission destination" and the evaluation operation (10) "selection of the OK button" with the right hand.

FIG. 12 is a diagram showing an example of the internal operation when the operation and the signal of FIG. 11 in the information processing apparatus 200 are detected. In the first step, the information processing apparatus 200 detects the evaluation operations (1) to (6). The information processing apparatus 200 records a series of operations 1201 including the evaluation operations (1) to (6) in the evaluation operation temporary data 1206.

In the second step, the information processing apparatus 200 detects the correction operation (7) and the insertion start signal (X1). Since the correction operation (7) and the insertion start signal (X1) are not the operations to be evaluated, the information processing apparatus 200 does not record the correction operation (7) and the insertion start signal (X1) in the evaluation operation temporary data 1206. Further, the information processing apparatus 200 recognizes that the subsequent evaluation operations are operations that are subject to the insertion process.

In the third step, the information processing apparatus 200 detects the evaluation operations (A) to (C). Since the evaluation operations (A) to (C) are operations after the insertion start signal (X1), the information processing apparatus 200 targets a series of operations 1203 including the evaluation operations (A) to (C) in the insertion process. The series of operations 1203 is recorded in the insertion operation temporary data.

In the fourth step, the information processing apparatus 200 detects the insertion end signal (X2) and the correction operation (8). Since the insertion end signal (X2) and the correction operation (8) are not the operations to be evaluated, the information processing apparatus 200 does not record the insertion end signal (X2) and the correction operation (8) in the evaluation operation temporary data 1206. Further, the information processing apparatus 200 recognizes that the subsequent evaluation operations are normal evaluation operations.

In the fifth step, since the range of the series of operations 1203 (insertion operation temporary data) to be the target of the insertion process is determined, the information processing apparatus 200 inserts the series of operations 1203 into the evaluation operation temporary data. At this time, a series of operations 1203 is inserted between the evaluation operation (3) and the evaluation operation (4). The information processing apparatus 200 can store the screen transitions before and after the insertion start signal (X1), and can determine the insertion position of the series of operations 1203 based on the screen transitions.

More specifically, first, the information processing apparatus 200 detects from the operation record that the insertion start signal (X1) has been executed on the home menu screen 1101. Next, the information processing device 200 detects from the operation record that the scan setting screen 1103 is displayed immediately before the home menu screen 1101. Subsequently, the information processing apparatus 200 acquires the evaluation operations (4) to (6) executed on the scan setting screen 1103 from the operation record. Finally, the information processing apparatus 200 determines the insertion position of the series of operations 1103 to the position immediately before the evaluation operations (4) to (6). That is, the information processing apparatus 200 identifies the screen (N) displayed immediately before the screen (M) on which the insertion start signal (X1) is executed, and the evaluation operation executed on the screen (N). A series of operations (O2) to be inserted is inserted at the start position of (O1).

In the sixth step, the evaluation operations (9) and (10) are detected. The information processing apparatus 200 records a series of operations 1205 including the evaluation operations (9) and (10) in the evaluation operation temporary data 1206.

As described above, the information processing apparatus 200 inserts a series of operations to be input processing at the start position of the evaluation operation of the screen displayed immediately before the screen on which the insertion start signal (X1) is executed. By this process, the user can later insert the initial setting operation at an appropriate position in the evaluation operation temporary data even when the operation includes a plurality of screen transitions. As a result, even if the user notices an input omission after performing a test input that repeats screen transitions many times, it is not necessary to restart the operation from the beginning, and the test man-hours of the device under test can be reduced. ..

[About the internal processing flow of the information processing device 200]
Next, the flow of internal processing of the information processing apparatus will be described with reference to FIGS. 13 and 14. FIG. 13 is a diagram showing an example of a procedure for generating an automatic execution script in the information processing apparatus 200. In a certain aspect, the CPU 301 may read the program for performing the process of FIG. 13 from the secondary storage device 303 into the primary storage device 302 and execute the program. In other aspects, some or all of the processing may also be realized as a combination of circuit elements configured to perform the processing.

In step S1310, the CPU 301 determines whether or not the operation mode has been set. The CPU 301 can accept the setting of the operation mode from the user via the input interface 305 or the like. The setting of the operation mode includes at least one of the operation mode of the evaluation operation, the operation mode of the correction operation, the mode of the insertion start signal, and the mode of the insertion end signal. In a certain aspect, the mode of the insertion start signal and the mode of the insertion end signal may be the same. Further, in another aspect, the setting of the operation mode may include the mode of the insertion start operation and the mode of the insertion end operation instead of the mode of the insertion start signal and the mode of the insertion end signal. When the CPU 301 determines that the operation mode has been set (YES in step S1310), the CPU 301 shifts control to step S1320. If not (NO in step S1310), the CPU 301 executes the process of step S1310 again.

In step S1320, the CPU 301 controls the camera 250 and the microphone 260, and starts recording the operation of the user's image forming apparatus 100. In step S1330, the CPU 301 acquires an operation record from the camera 250. The operation record may be a video or an image. Further, the operation record may include sound information acquired from the microphone 260. The CPU 301 updates the evaluation operation temporary data based on the operation record. In some aspects, the operation record may include at least one of an evaluation operation, a correction operation, an insertion start signal and an insertion end signal. Also, in other aspects, the operation record may include at least one of an evaluation operation, a correction operation, an insertion start operation and an insertion end operation.

In step S1340, the CPU 301 executes the operation information recording process. The operation information recording process is a process for generating evaluation operation temporary data based on the evaluation operation, the correction operation, and the signal described with reference to FIGS. 5 to 12. The details of the operation information recording process will be described with reference to FIG.

In step S1350, the CPU 301 determines whether or not to end the recording of the operation of the user's image forming apparatus 100. As an example, the CPU 301 may accept an instruction to end recording of the operation of the user's image forming apparatus 100 via the input interface 305 or the like. When the CPU 301 determines that the recording of the operation of the user's image forming apparatus 100 is finished (YES in step S1350), the CPU 301 shifts the control to step S1360. If not (NO in step S1350), the CPU 301 shifts control to step S1330. In step S1360, the CPU 301 generates an automatic execution script based on the evaluation operation temporary data.

FIG. 14 is a diagram showing an example of an execution procedure of the operation information recording process in the information processing apparatus 200. In a certain aspect, the CPU 301 may read the program for performing the process of FIG. 14 from the secondary storage device 303 into the primary storage device 302 and execute the program. In other aspects, some or all of the processing may also be realized as a combination of circuit elements configured to perform the processing.

In step S1405, the CPU 301 determines whether or not the user's operation is a mode switching operation. The "mode switching operation" here corresponds to an insertion start signal or an insertion end signal. That is, the CPU 301 determines whether or not a signal set in the information processing apparatus 200 (such as smoothing a finger, clapping a hand, or issuing a specific word) is detected as an insertion start signal or an insertion end signal. In a certain aspect, the "mode switching operation" may be an insertion start operation or an insertion end operation.

When the CPU 301 determines that the user's operation is a mode switching operation (YES in step S1405), the CPU 301 shifts the control to step S1410. If not (NO in step S1405), the CPU 301 shifts control to step S1430.

In step S1410, the CPU 301 determines whether or not the current operation mode of the information processing device 200 is the insertion mode. The "insertion mode" here is an operation mode until the CPU 301 detects the insertion start signal (X1) in FIG. 9 and the like and detects the insertion end signal (X2).

When the CPU 301 determines that the current operation mode of the information processing apparatus 200 is the insertion mode (YES in step S1410), the CPU 301 shifts control to step S1415. If not (NO in step S1410), the CPU 301 shifts control to step S1425.

In step S1415, the CPU 301 turns off the insertion mode. This step is a process when the CPU 301 detects a signal in step S1405 while the information processing device 200 is operating in the insertion mode. In this case, the signal detected in step S1405 becomes the insertion end signal. Therefore, the CPU 301 changes the current operation mode of the information processing apparatus 200 from the insert mode to the normal mode.

In step S1420, the CPU 301 inserts the insert operation temporary data temporarily stored in the insert operation store 215 into the evaluation operation temporary data. The CPU 301 executes the insertion process based on the procedure described with reference to FIGS. 9 to 12.

In step S1425, the CPU 301 turns on the insertion mode. This step is a process when the CPU 301 detects a signal in step S1405 while the information processing device 200 is operating in the normal mode. In this case, the signal detected in step S1405 becomes the insertion start signal. Therefore, the CPU 301 changes the current operation mode of the information processing apparatus 200 from the normal mode to the insert mode.

In step S1430, the CPU 301 determines whether or not the user's operation is a correction operation. When the CPU 301 determines that the user's operation is a correction operation (YES in step S1430), the CPU 301 shifts control to step S1435. If not (NO in step S1430), the CPU 301 shifts control to step S1440. In step S1435, the CPU 301 saves the user's operation as a correction operation in the correction operation store 207.

In step S1440, the CPU 301 determines whether or not the current operation mode of the information processing apparatus 200 is the insertion mode. When the CPU 301 determines that the current operation mode of the information processing apparatus 200 is the insertion mode (YES in step S1440), the CPU 301 shifts control to step S1445. If not (NO in step S1440), the CPU 301 shifts control to step S1450.

In step S1445, the CPU 301 saves the user's operation in the insertion operation store 215 as an operation to be inserted. The operations targeted for the insertion process here are, for example, operations (A) to (C) in FIG. 9 and the like. In step S1450, the CPU 301 stores the user's operation as an evaluation operation in the evaluation operation store 208.

As described above, the information processing apparatus 200 according to the present embodiment recognizes a series of operations to be the target of the insertion process by the insertion start signal and the insertion end signal. Then, the information processing apparatus 200 inserts a series of operations to be inserted at the beginning of the evaluation operation temporary data, and generates an automatic execution script based on the evaluation operation temporary data. By this process, the user does not have to restart the operation from the beginning when the user forgets the initial setting during the operation of the device to be tested, and the man-hours for creating the automatic execution script can be reduced.

Further, the information processing apparatus 200 can later insert the initial setting operation into an appropriate position of the evaluation operation temporary data based on the recording of the screen transition even in the operation of transitioning a plurality of screens. By this process, even if the user notices an input omission after performing a test input in which the screen transition is repeated many times, it is not necessary to restart the operation from the beginning, and the test man-hours of the device to be tested can be reduced.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the scope of claims, and it is intended that all modifications are included in the meaning and scope equivalent to the scope of claims. In addition, the inventions described in the embodiments and the modifications are intended to be implemented, either alone or in combination, wherever possible.

100 image forming device, 101 control unit, 102 document reading unit, 103 print output unit, 104 FAX unit, 105 storage device, 106 wireless interface, 107 wired interface, 108 user authentication unit, 109 operation panel, 110, 301 CPU, 111 RAM, 112 ROM, 113 display unit, 114 operation unit, 200 information processing device, 201 screen analysis unit, 202 operation analysis unit, 203 operation judgment unit, 204 correction processing unit, 205 script generation unit, 206 content DB, 207 correction operation Store, 208 Evaluation Operation Store, 209 Script DB, 210,601 Operation Record, 215 Insert Operation Store, 250 Camera, 260 Microphone, 302 Primary Storage, 303 Secondary Storage, 304 External Device Interface, 305 Input Interface, 306 Output interface, 307 communication interface, 401 screen, 501,701,1101 home menu screen, 502,1102 copy setting screen, 503 margin setting screen, 602,802,1006,1206 evaluation operation temporary data, 603 correction operation temporary data, 702 , 1103 Scan setting screen, 703, 1104 Initial setting screen, 810,820,1001,1003,1005,1201,1203,1205 A series of operations.

Claims (12)

A method for test automation of equipment,
A step to detect the user's operation on the operation screen of the target device that accepts the user's operation input, and
Including a step of determining a plurality of user operation modes in the operation.
The plurality of operation modes
The first operation mode registered in the operation record and
Including a second operation mode not registered in the operation record.
The method is
A step of recording a first series of operations including a screen transition performed by the first operation mode, and
A step of accepting one or more operations including a screen transition executed by the second operation mode, and
The step of accepting the start signal of the insert operation and
A step of accepting a second series of operations according to the first operation mode, and
The step of accepting the end signal of the insert operation and
A method further comprising the step of generating an operation record assuming that the second series of operations is an operation received prior to the first series of operations. The first series of operations includes a screen transition from the first screen to the second screen.
One or more operations including the screen transition executed by the second operation mode include the operation of the screen transition from the second screen to the first screen.
The step of accepting the start signal of the insertion operation is executed on the first screen.
One or more operations including the screen transition executed by the second operation mode include a screen transition from the first screen to the third screen and a screen from the third screen to the first screen. Including transitions
The method according to claim 1, wherein the step of receiving the end signal of the insertion operation is executed on the first screen. The step of generating the operation record is the case where a third series of operations including a screen transition from the first screen to the fourth screen is accepted before the first series of operations. The method according to claim 2, wherein the operation record is generated assuming that the operations are received in the order of the third series of operations, the second series of operations, and the first series of operations. The step of accepting the start signal of the insert operation and the step of accepting the end signal of the insert operation are respectively.
Steps to accept sound input via a microphone,
The method according to any one of claims 1 to 3, comprising a step of determining whether or not the sound input is an input corresponding to the disclosure signal or the end signal of the insertion operation. The step of accepting the start signal of the insert operation and the step of accepting the end signal of the insert operation are respectively.
Steps to capture the movement of the user's hand through the camera,
The method according to any one of claims 1 to 4, comprising a step of determining whether or not the movement of the user's hand corresponds to the disclosure signal or the end signal of the insertion operation. The step of determining the plurality of operation modes is based on whether the finger touching the operation screen is either the finger of the right hand or the finger of the left hand, and each operation mode is the first operation mode or the second operation mode. The method according to any one of claims 1 to 5, which includes a step of determining which of the operation modes of the above. The step of determining the plurality of operation modes determines whether each operation mode is either the first operation mode or the second operation mode based on the shape or color of the touch pen that touches the operation screen. The method of any of claims 1-5, which comprises steps. The step of determining the plurality of operation modes is a step of determining whether each operation mode is the first operation mode or the second operation mode based on the shape of the object touching the operation screen. The method according to any one of claims 1 to 5, including the method according to any one of claims 1 to 5. A method for test automation of equipment,
A step to detect the user's operation on the operation screen of the target device that accepts the user's operation input, and
Including a step of determining a plurality of user operation modes in the operation.
The plurality of operation modes
The first operation mode registered in the operation record and
Including a second operation mode and a third operation mode that are not registered in the operation record.
The method is
A step of recording a first series of operations including a screen transition performed by the first operation mode, and
A step of accepting one or more operations including a screen transition executed by the second operation mode, and
The step of accepting the insertion start operation according to the third operation mode, and
A step of accepting a second series of operations according to the first operation mode, and
The step of accepting the insertion end operation according to the third operation mode, and
A method further comprising the step of generating an operation record assuming that the second series of operations is an operation received prior to the first series of operations. The first series of operations includes a screen transition from the first screen to the second screen.
One or more operations including the screen transition executed by the second operation mode include the operation of the screen transition from the second screen to the first screen.
The insertion start operation is a screen transition from the first screen to the third screen.
One or more operations including the screen transition executed by the second operation mode are one or more operations on the third screen.
The method according to claim 9, wherein the insertion end operation is a screen transition from the third screen to the first screen. A program that causes a computer to execute the method according to any one of claims 1 to 10. With the processor
An apparatus comprising a memory for storing a program for causing the processor to execute the method according to any one of claims 1 to 10.

Images