JP4896909B2 - Scenario generation apparatus and program - Google Patents

Description

  The present invention relates to a scenario generation device and a scenario generation program that generate a scenario based on a specified image. For example, the specification of an image that causes an error during scenario execution can be prevented, and the efficiency of scenario correction can be improved. The present invention relates to a scenario generation device and a program.

  In general, an information system in which a CPU executes software is widely known. In this type of information system, when a large amount of processing requests occur, the load on the CPU increases in response to processing requests. The performance of the information system at such a high load is extremely important. Specifically, resource (CPU etc.) utilization rate at high load, response time at high load, and the like are important performances for information system users.

  When developing and operating an information system, from the viewpoint of measuring and verifying performance at high loads, load tests for measuring the performance of information systems are widely conducted in the simulated high load situation. ing. In a load test, generally, a large amount of processing requests are generated in a pseudo manner from a machine corresponding to an information terminal (see, for example, Patent Document 1). A machine corresponding to an information terminal usually generates a load by executing an installed load test program. The CPU that executes the load test program generates from the information terminal a processing request similar to the processing request when a human operates and inputs the information terminal.

  Here, there are several types of load test programs. A relatively popular method is a method in which processing requests sent from the client software to the server are recorded in advance, and the pre-recorded processing requests are repeatedly generated in a pseudo manner from the load generation tool to the server.

  However, in this system, in a complex system in which processing requests change every time, it is difficult to provide a load generation tool with a mechanism for changing processing requests as in the case of client software. . Furthermore, from the viewpoint of appropriately changing processing requirements, when it is necessary to understand the internal behavior of client software, a high level of skill is required for the load tester. It becomes difficult.

  On the other hand, as another realization method, a method of generating a load by automatic operation of the screen of the client software can be considered. This is because the client software is activated on the information terminal, and the load test tool performs a similar operation as a human operation to cause the client software to operate, thereby generating a processing request from the client software. Is sent to the server.

  In the case of this method, an input event (operation event) from a mouse or a keyboard is input to a predetermined operation target such as a button in accordance with the screen transition of the client software. Such automatic operation of the screen is defined in advance in the form of an operation scenario (hereinafter also simply referred to as a scenario). In this method, since the processing request is generated by using the mechanism installed in the client software as it is, a mechanism for changing the processing request every time is not required unlike the above. Furthermore, since a person who understands how to operate the client software can create an operation scenario, advanced skills are not required.

  In such a load generation method by automatic screen operation, an operation target such as a button or a text box displayed in a window on the information terminal is used. When an operation event is input to this operation target, several realization methods can be considered.

  In the first implementation method, an ID for identifying the operation target in the screen is determined based on a mechanism in which the software manages the operation target in the screen by understanding the structure of the software constituting the screen. Clearly, the ID and the type of event to be input to the ID are described in the operation scenario. When the scenario is executed, a mouse click event or the like is input to the operation target based on the operation target ID.

  However, in the first implementation method, when the management method of the operation target by software changes, the method of specifying the operation target and the method of specifying the display position change. For this reason, since it is necessary to develop a load test program corresponding to the management method of the operation target by each software, a great cost is generated.

  On the other hand, in the second realization method, the operation target on the screen is compared with the image information acquired in advance, the display position of the operation target on the screen is specified, and the mouse is moved to the display position of the operation target. To generate an event corresponding to an operation such as clicking. According to the second realization method, since the operation target is specified based on the appearance (shape, color, etc.) of the operation target, it is not necessary to develop a load test program corresponding to the management method of the operation target.

As software for the second realization method, for example, an auto mouse published as free software is known (for example, see Non-Patent Document 1).
JP 2003-208329 A Masaya Yama: Programming method using GUI as API, Transactions of Information Processing Society of Japan: Programming Vol. 39, no. SIG 1 (PRO 1), pp. 26-33, (1998).

  However, in the second realization method as described above, there is a problem that labor for creating an operation scenario is large when the client software having various screens is used efficiently.

  For example, in the conventional method, image information as a snapshot of the screen of an information terminal (PC) is acquired, a part of an image to be operated is specified using a special editor, and the operation target is It was necessary to define what input events should be thrown from the mouse and keyboard. For a type of client software that sequentially switches a plurality of operation screens, it takes a lot of labor to create an operation scenario in such a procedure, which has been a factor of significantly reducing work efficiency.

  Therefore, the present inventors have collected and recorded events that occur in accordance with the operation in Japanese Patent Application No. 2007-196422 that has not been published at the time of filing of this application, and operated the entire operation screen on the operation screen. A third realization method has been proposed in which an operation scenario is efficiently created from these events and each image by collecting and recording the images designated as targets.

  However, in the third realization method, an unskilled operator may create a scenario by specifying an image that causes an error when executing the scenario. When such a scenario is generated, an error does not occur until the scenario is executed, so that it is difficult to correct an image that causes the error. In addition, for an unskilled operator, when an error occurs, the cause of the error is not clearly understood, and there is a concern that the work efficiency of scenario correction is reduced. In addition, it takes time to restart the client application to find the cause of the error, and there is a concern that the work efficiency may be further reduced.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a scenario generation device and a program that can prevent the designation of an image that causes an error during scenario execution and improve the efficiency of scenario correction.

  The first aspect of the present invention describes a procedure for generating an operation event for a comparison target image including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. A scenario generation device for generating an operation scenario, wherein event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen, and when the event information is an instruction of an extraction range including position coordinates, Based on the event information, extraction range determining means for determining extraction range information indicating a range from which the comparison target image is extracted; and based on the extraction range information, the extraction range of the comparison target image is displayed on the operation screen. When the extraction range display means to display and the event information is a recording instruction following the extraction range instruction, the operation screen is displayed based on the event information. Image acquisition means for extracting and storing the image information to be compared from the image information of the entire image based on the coordinate value of the image information of the entire image and the extraction range information, the stored event information, and the A scenario generation apparatus comprising scenario generation means for generating the operation scenario based on image information to be compared.

  The second aspect of the present invention describes a procedure for generating an operation event for a comparison target image including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. A scenario generation device for generating an operation scenario, wherein event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen, and when the event information is an instruction of an extraction range including position coordinates, Based on the event information, extraction range determining means for determining extraction range information indicating a range for extracting the comparison target image; and when the event information is a recording instruction following the extraction range instruction, Then, based on the coordinate value of the image information of the whole image on the operation screen and the extraction range information, the image information of the whole image is compared with the comparison target. Image acquisition means for extracting and storing image information, and scenario generation means for generating the operation scenario based on the stored event information and the image information to be compared, and as the image acquisition means, When a plurality of window screens are displayed on the operation screen, based on the recording instruction, window range storage means for storing window range information and overlap order information indicating the range of each window screen and the overlap order, respectively, Based on the position coordinates included in the instruction of the extraction range, the window range information, and the overlapping order information, window range information corresponding to the overlapping order information indicating the highest position among the window range information including the position coordinates is obtained. Window selection means to be selected and window range information selected by the window selection means , Common range extraction means for extracting common range information indicating a common range with the extraction range information, extraction range change means for changing the value of the extraction range information to the value of the common range information, and the changed value And an image extraction storage means for extracting and storing the image information to be compared based on the extracted range information.

  The third aspect of the present invention describes a procedure for generating an operation event for a comparison target image including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. A scenario generation device for generating an operation scenario, which is extracted from event information generated by an operation on the operation screen, image information of an entire image on the operation screen, and the operation screen for a certain step number The storage means for storing the operation scenario in which the comparison target image information is associated with each other, and the image information of the entire image and the comparison target image information for each step number are extracted from the operation scenario in the storage means, respectively. Image extracting means, and a ratio extracted by the image extracting means in the image information of the whole image extracted by the image extracting means. Determination means for determining whether the image information of the object can be uniquely determined, the result of the determination, if not, a scenario generation device including a result display means for displaying that the relevant negative, the.

  Each aspect of the present invention is expressed as an apparatus, but is not limited thereto, and can be expressed as a program, a computer-readable storage medium storing the program, or a method.

(Function)
According to the first aspect, with the configuration in which the extraction range of the comparison target image is displayed on the operation screen, it is possible to prevent the designation of an image that causes an error during scenario execution, and to improve the efficiency of scenario correction.

  According to the second aspect, the position indicated by the configuration for extracting the image to be compared based on the common range of the top-level window range information including the position coordinates included in the extraction range instruction and the extraction range information. Since a background window screen that does not include coordinates is no longer extracted as part of the image to be compared, it is possible to prevent the designation of an image that causes an error during scenario execution, and to improve the efficiency of scenario correction.

  According to the third aspect, it is determined whether or not the extracted image information to be compared can be uniquely determined. Specification can be prevented and the work efficiency of scenario correction can be improved.

  As described above, according to the present invention, it is possible to prevent the designation of an image that causes an error during scenario execution, and to improve the work efficiency of scenario correction.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each of the following devices can be implemented for each device with either a hardware configuration or a combination configuration of hardware resources and software. As the software of the combined configuration, a program that is installed in advance on a computer of a corresponding device from a network or a storage medium and that realizes the function of the corresponding device is used.

(First embodiment)
First, a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram showing the configuration of the entire system for performing a load test, and includes a plurality of information terminals (PCs) 1, 1,... On the client side that generate a load and a server 2 to be subjected to the load test. Composed. A plurality of information terminals (PCs) 1, 1,... On the client side are provided with a load generation unit that uses the load generation system 10 and the client application execution unit 20 that make processing requests to the server 2 as unit load generation sources. It is done.

  In the load test, by adding a large amount of processing requests to the server 2 from the load generating means provided in the plurality of information terminals (PCs) 1, 1,. (CPU etc.) Measure the utilization rate, response time for processing requests, etc.

The configuration of the load generation system 10 shown in FIG. 1 is shown in FIG.
The load generation system 10 generates a mouse event or a keyboard event to a client application execution unit 20 in which a CPU executes a client application program in a memory. I do. The client application program is accompanied by a screen operation using a GUI (Graphical User Interface). The client application execution unit 20 issues a processing request to the server 2 in response to a screen operation from the load generation system 10.

  As illustrated in FIG. 2, the load generation system 10 includes a control unit 11, a screen transition determination unit 12, a comparison target identification unit 13, an operation event generation unit 14, and an image information acquisition unit 15.

  In the load generation system 10 illustrated in FIG. 2, the control unit 11 includes an operation execution control unit 111 and a scenario storage unit 112.

  The operation execution control unit 111 provided in the control unit 11 controls the screen transition determination unit 12, the comparison target specifying unit 13, and the operation event generation unit 14 according to the description contents of the operation scenario in the scenario storage unit 112. The scenario storage unit 112 stores an operation scenario that defines the contents of a series of screen operations to be executed by the load generation system 10 by operations on the screen.

  The operation execution control unit 111 provided in the control unit 11 issues a screen transition determination request 213 to the screen transition determination unit 12, receives a screen transition determination response 221 from the screen transition determination unit 12, and specifies a comparison target. A function for accepting a comparison target identification response 222 from the comparison target identification unit 13 and an event generation request 212 for the operation event generation unit 14. 14 is provided with a function of accepting an event occurrence response 223 from 14.

  The screen transition determination unit 12 issues a screen information acquisition request 215 to the image information acquisition unit 15 in accordance with the screen transition determination request 213 received from the operation execution control unit 111, and sends an image information acquisition response 225 from the image information acquisition unit 15. It has a function of accepting and sending a screen transition determination response 221 to the operation execution control unit 111.

  The comparison target specifying unit 13 issues a screen information acquisition request 216 to the image information acquisition unit 15 in accordance with the comparison target specification request 211 received from the operation execution control unit 111, and sends an image information acquisition response 226 from the image information acquisition unit 15. It has a function of accepting and sending the comparison target specifying response 222 to the operation execution control unit 111.

  The operation event generation unit 14 has a function of issuing an event 201 to the client application execution unit 20 in accordance with an event generation request 212 received from the operation execution control unit 111 and sending an event generation response 223 to the operation execution control unit 111. ing. The client application execution unit 20 sends a processing request 202 according to the event 201 received from the operation event generation unit 14 to the server 2.

  The image information acquisition unit 15 acquires the image information 203 from the client application execution unit 20 in accordance with the screen information acquisition request 215 received from the screen transition determination unit 12, and sends the image information acquisition response 225 to the screen transition determination unit 12. And a function of acquiring image information 203 from the client application execution unit 20 in accordance with the screen information acquisition request 216 received from the comparison target specifying unit 13 and sending an image information acquisition response 226 to the comparison target specifying unit 13.

  The comparison target specifying request 211, the screen transition determination request 213, and the image information acquisition responses 225 and 226 each include image information of a bitmap image (file with extension bmp) included in the operation screen of the client application execution unit 20. Accompanying. However, the image file is not limited to a bitmap image, and may be an image file of another format.

  The configuration of the comparison target specifying unit 13 shown in FIG. 2 is shown in FIG.

  As illustrated in FIG. 3, the comparison target specifying unit 13 includes a comparison target specifying control unit 131 and an image comparison unit 133.

  In accordance with the comparison target specification request 211 received from the operation execution control unit 111, the comparison target specification control unit 131 sends the image information of the bitmap image associated with the request 211 to the image comparison unit 133 and the screen information acquisition request 216. Is sent to the image information acquisition unit 15 to acquire image information of a bitmap image accompanying the image information acquisition response 226 from the image information acquisition unit 15 and supply the image information to the image comparison unit 133. Realize.

  The image comparison unit 133 searches the screen information on the operation screen of the client application execution unit 20 for a portion that matches the bitmap image specified by the comparison target specification request 211. If a matching part is found as a result of the search, a search result notification is sent to the comparison target specifying control unit 131. The search result notification includes the coordinate information of the portion that matches the bitmap image in the screen information. Note that the term “match” as used herein may mean that all the pixels constituting the bitmap image are completely matched, or may be matched not less than a specified standard, though not completely. . For example, “match” may be defined on the condition that the pixel color information matches at a rate equal to or greater than a predetermined value.

  Here, the operation of the load generation system 10 will be described with reference to FIGS. 4 to 11 by giving a specific example of the operation screen in the client application execution unit 20. Here, the operation will be described taking the schedule display screen shown in FIG. 4 as an example of the application operation window handled by the client application execution unit 20.

  The operation execution control unit 111 provided in the control unit 11 of the load generation system 10 includes a screen transition determination unit 12 provided in the load generation system 10 and a comparison target specification according to the definition of the operation scenario in the scenario storage unit 112. Each component such as the unit 13, the operation event generation unit 14, and the image information acquisition unit 15 is controlled.

  Under the control unit 11, the operation execution control unit 111 sends a comparison target specifying request 211 to the comparison target specifying unit 13 according to the contents described (defined) in the operation scenario in the scenario storage unit 112. An event generation request 212 is sent to the event generation unit 14, and a screen transition determination request 213 is sent to the screen transition determination unit 12.

  The screen transition determination unit 12 and the comparison target identification unit 13 send screen information acquisition requests 215 and 216 to the image information acquisition unit 15 under the control of the operation execution control unit 111, respectively. That is, when the screen transition determination unit 12 receives the screen transition determination request 213 from the operation execution control unit 111, the screen transition determination unit 12 sends a screen information acquisition request 215 to the image information acquisition unit 15. When receiving the comparison target specification request 211 from the operation execution control unit 111, the comparison target specifying unit 13 sends a screen information acquisition request 216 to the image information acquisition unit 15.

  Upon receiving the screen information acquisition request 215 from the screen transition determination unit 12, the image information acquisition unit 15 acquires the image information 203 displayed on the screen by the client application execution unit 20 via the operating system (OS) execution unit 40. The image information acquisition response 225 including the acquired image is sent to the screen transition determination unit 12. Similarly, when the screen information acquisition request 216 is received from the comparison target specifying unit 13, the client application execution unit 20 acquires the image information 203 displayed on the screen, and the image information acquisition response 226 including the acquired image is specified as the comparison target. Send to unit 13.

  A specific example of the image information (application operation window) 203 displayed on the screen by the client application execution unit 20 at this time is shown in FIG. Here, a schedule display screen 21 in the schedule management system is shown as an example. In the subsequent operation description, the image information on the schedule display screen 21 shown in FIG. 4 is acquired as the image information 203 by the image information acquisition unit 15, and the image information on the schedule display screen 21 is the screen transition determination unit 12 and the comparison. It is assumed that it is sent to the target specifying unit 13.

  When the button “travel expense clearing system” is clicked on the schedule display screen 21 shown in FIG. 4, a transition is made to the travel expense clearing system screen 22 shown in FIG. 5. Further, when the “collective billing” button is clicked on the screen 22 of this travel expense clearing system, the screen further transitions to the next screen.

  A procedure for realizing the operation corresponding to the series of screen transitions by the load generation system 10 will be described below.

  The coordinate axes of the schedule display screen 21 shown in FIG. 4 are shown in FIG. Here, an XGA (1024 × 768 pixels) application operation window is shown as an example.

  A partial description example of the operation scenario in the scenario storage unit 112 is shown in FIG. In this operation scenario, as shown in FIG. 7, an operation step, screen identification information (screen ID), a screen transition determination image of a bitmap image, a comparison target image, determination / operation contents, and numerical information are set. Configured as an item (definition item).

  The client application execution unit 20 is in a state of displaying the schedule display screen 21 shown in FIG. 4 during the screen transition.

  The operation execution control unit 111 reads step 20 of the operation scenario in the scenario storage unit 112. In this step 20, “screen transition determination” is set (defined) as [determination / operation content]. Accordingly, it is determined that the next operation is to determine screen transition. Further, the file name “1.bmp” is set as the [screen transition determination image]. The contents (bitmap image) are shown in FIG. The operation execution control unit 111 sends a screen transition determination request 213 including information (bitmap image) of the screen transition determination image shown in FIG. 8 to the screen transition determination unit 12.

  Upon receiving the screen transition determination request 213 from the operation execution control unit 111, the screen transition determination unit 12 sends a screen information acquisition request 215 to the image information acquisition unit 15.

  Upon receiving the screen information acquisition request 215 from the screen transition determination unit 12, the image information acquisition unit 15 acquires the image information (bitmap image) 203 displayed by the client application execution unit 20 and includes the image information 203. An image information acquisition response 225 is sent to the screen transition determination unit 12.

  The screen transition determination unit 12 checks whether or not there is a portion that matches the “1.bmp” image in the image information 203 acquired from the client application execution unit 20 via the image information acquisition unit 15. The contents (bitmap image) of “1.bmp” at this time are information (bitmap image) of the screen transition determination image shown in FIG.

  In this examination, for example, if the matching part does not exist because the screen transition is not completed, the screen transition determination unit 12 sends the screen information acquisition request 215 to the image information acquisition unit 15. Therefore, the same processing as described above is repeatedly executed at an appropriate frequency.

  When the screen transition is completed and a matching part is found, the screen transition determination unit 12 sends a screen transition determination response 221 to the operation execution control unit 111.

  The operation execution control unit 111 reads step 21 which is the next step of the operation scenario in the scenario storage unit 112. In this step 21, “click” is set as [determination / operation content]. Thus, the next thing to do is determined to be a click. In addition, the file name “2.bmp” is set as the “comparison target image”. The bitmap image of “2.bmp” is shown in FIG.

  The operation execution control unit 111 sends a comparison target specifying request 211 to the comparison target specifying unit 13 according to the contents of the operation scenario in the scenario storage unit 112. This comparison target specifying request 211 is accompanied by the comparison target image (bitmap image) shown in FIG.

  In the comparison target specifying unit 13, when receiving the comparison target specifying request 211 from the operation execution control unit 111, the comparison target specifying control unit 131 sends a screen information acquisition request 216 to the image information acquisition unit 15.

  When receiving the screen information acquisition request 216 from the comparison target specifying control unit 131 provided in the comparison target specifying unit 13, the image information acquiring unit 15 acquires the image information 203 displayed by the client application executing unit 20, The acquired image information 203 is sent to the image comparison unit 133 provided in the comparison target specifying unit 13. In this case, the schedule display screen 21 shown in FIG. 4 is sent to the image comparison unit 133 as the image information 203.

  The image comparison unit 133 is an image (bitmap image) represented by the file name “2.bmp” as a comparison target image from the image information 203 acquired from the client application execution unit 20 via the image information acquisition unit 15. The part that matches 112 is searched.

  As a result of the search, the image comparison unit 133 responds with [center X coordinate] and [center Y coordinate] to the comparison target specifying control unit 131, and the comparison target specifying control unit 131 compares the operation execution control unit 111 with the comparison. A target identification response 222 is sent.

  The operation execution control unit 111 uses the comparison target identification response 222 received from the comparison target identification control unit 131 as information on the comparison target center X coordinate and center Y coordinate, and information on “click” as the operation / determination content. In association with the generation request 212, the event generation request 212 is sent to the operation event generation unit 14.

  In accordance with the event generation request 212 received from the operation execution control unit 111, the operation event generation unit 14 generates an event 201 equivalent to the movement of the mouse to the center X coordinate and the center Y coordinate to be compared. Further, a mouse click event 201 is generated. Thereafter, an event occurrence response 223 is sent to the operation execution control unit 111.

  In this way, an appropriate operation event can be generated such that the screen of the client application execution unit 20 is operated.

  Next, the operation execution control unit 111 reads Step 22, which is the next step of the operation scenario in the scenario storage unit 112. In this step 22, “screen transition determination” is set as [determination / operation content].

  Here, the processing in step 22 is executed in the same manner as in step 20 described above. The contents (bitmap image) of “3.bmp” in step 22 are shown in FIG.

  The operation execution control unit 111 reads step 23 which is the next step of the operation scenario in the scenario storage unit 112.

  Here, the processing in step 23 is executed in the same manner as in step 21 described above. The contents (bitmap image) of “4.bmp” in step 23 are shown in FIG.

  In this way, an appropriate operation event can be generated on the screen of the client application execution unit 20.

  By the processing operation according to the operation scenario in the scenario storage unit 112 described above, the load generation system 10 can cause the client application execution unit 20 to generate appropriate events sequentially in accordance with the screen transition. A load test for applying a desired load to the server 2 by sending a processing request from the client application execution unit 20 to the server 2 becomes possible.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. The second embodiment realizes a system for creating an operation scenario used in the first embodiment for an application program with screen transition.

  In the second embodiment of the present invention, an operation describing a procedure for generating an operation event for a comparison target including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. A scenario generation system for generating a scenario, wherein an event acquisition unit that collects and records an event that occurs in response to an operation on the operation screen, an entire image of the operation screen, and an extracted image of a specified part on the operation screen Image acquisition means that collects and records the event, and a scenario generation means that generates the operation scenario based on the event acquired by the event acquisition means and the image acquired by the image acquisition means. The event acquisition means is realized by an event detection unit 308, an event recording unit 309, and an event log storage unit 306 shown in FIG. The image acquisition means is realized by a screen image recording instruction unit 311 and a screen image recording unit 310 shown in FIG. 13 to be described later, the entire screen image information recorded by the screen image recording unit 310, and a comparison target range extracting unit 303. The The scenario generation means is realized by a scenario generation unit 302 shown in FIG.

  FIG. 12 shows the configuration of the entire system for creating scenarios according to the second, third, and fourth embodiments of the present invention. The system shown in FIG. 12 is realized with the scenario generation system 30 as a main component. The scenario generation system 30 sends various operation events generated via the operating system (OS) execution unit 40 to the client application execution unit 20 that sends a processing request to the server 2 in response to an operation instruction from the operator 3. A function of creating an operation scenario and recording it in the scenario storage unit 112 is realized along with the image information of the client application execution unit 20.

The configuration of this scenario generation system 30 is shown in FIG.
The scenario generation system 30 illustrated in FIG. 13 includes a scenario generation unit 302, a comparison target range extraction unit 303, an event detection unit 308, an event recording unit 309, a screen image recording unit 310, a screen image recording instruction unit 311, and the like. Is done.

  The screen image recording instruction unit 311 is operated by the operator 3 when the client application execution unit 20 captures an image displayed on the screen, and is realized by, for example, a “PrintScreen” key on the keyboard.

  The screen image recording unit 310 captures the image information (entire screen image information) displayed on the screen by the client application execution unit 20 in accordance with the operation of the screen image recording instruction unit 311, and records it in the entire screen image information storage unit 307. . For example, all the images (bitmap images) on the schedule display screen 21 shown in FIG. 4 are recorded in a predetermined recording area (the entire screen image information storage unit 307) as entire screen image information. The “entire screen image” referred to here is an entire image within the operation target range, and may be an image within the operation target range including the position where the mouse operation will be performed. It includes the case of the image of the window that is.

  The event detection unit 308 detects operation event information generated through the operating system (OS) execution unit 40 in accordance with various operation instructions such as mouse click operation and key operation by the operator 3. This event information includes coordinate information indicating the position of the operation on the operation screen operated by the operator 3.

  The event recording unit 309 records various event information generated by the mouse click operation, key operation, and the like detected by the event detection unit 308 in the event log storage unit 306 as an event log.

  The comparison target range extraction unit 303 extracts image information (bitmap image) of the comparison target range according to the event log in the event log storage unit 306 from the entire screen image information in the entire screen image information storage unit 307 as comparison target image information. And recorded in the comparison target image information storage unit 301.

  The scenario generation unit 302 generates an operation scenario based on the comparison target image information in the comparison target image information storage unit 301 and the event log in the event log storage unit 306. Here, an operation scenario with the client application execution unit 20 as an operation target is created and recorded in the scenario storage unit 112.

  FIG. 14 shows a processing procedure of a system having the scenario generation system 30 in the second embodiment of the present invention as a main component, FIG. 7 shows an example of the contents of an operation scenario generated by the processing, An example of the set comparison target image information is shown in FIGS. Moreover, an example of the state transition of the comparison target image accompanying the operation of the mouse cursor on the operation screen is shown in FIGS.

  Here, the operation scenario creation processing operation in the second embodiment of the present invention will be described with reference to FIGS.

  The operator 3 who is a scenario creator first activates the client application execution unit 20 which is an operation target. Then, the operation of the scenario generation system 30 is started.

  The operator 3 operates the client application execution unit 20 by sending various events such as mouse movement, mouse click, and keyboard input to the client application execution unit 20 via the operating system (OS) execution unit 40. To do.

  The scenario generation system 30 acquires information about the various events and image information of the client application execution unit 20 from the operating system (OS) execution unit 40 to generate an operation scenario and record it in the scenario storage unit 112.

  The second embodiment of the present invention shown in FIGS. 12 and 13 will be described below with reference to the processing procedure shown in FIG. 14, the contents of the operation scenario shown in FIG. 7, and the comparison target images shown in FIGS. A specific processing operation of a system in which the scenario generation system 30 in the embodiment is a main component will be described.

  In the processing flow shown in FIG. 14, in step S <b> 21, the operator 3 recognizes an intended comparison target from the image on the screen of the client application execution unit 20 and operates the screen image recording instruction unit 311. For example, when the schedule display screen 21 shown in FIG. 4 is displayed and the operator 3 finds a button for the “travel expense clearing system” in the lower left on the schedule display screen 21, the “PrintScreen” key on the keyboard, etc. Press the button.

  In step S <b> 22, the screen image recording instruction unit 311 instructs the screen image recording unit 310 to record the entire screen image information in accordance with the operation of the screen image recording instruction unit 311.

  In step S23, the screen image recording unit 310 follows the instruction from the screen image recording instruction unit 311, and the schedule display screen 21 shown in FIG. Recording is performed in a recording area (entire screen image information storage unit 307).

  In step S24, the operator 3 moves the mouse onto the “travel expense settlement system” button on the schedule display screen 21 and performs a click operation.

  In step S25, the event recording unit 309 detects the click operation of the mouse as an event, and coordinates information indicating the position of the mouse and event type information (click in this example) are used as an event log in a predetermined recording area ( Record in the event storage unit 306).

  In step S <b> 26, the comparison target range extraction unit 303 sets a predetermined value centered on coordinate information recorded as an event log in the event log storage unit 306 from the entire screen image information in the entire screen image information storage unit 307. A portion included in an area determined by the shape and size is extracted as a matching range, and is recorded in a predetermined recording area (comparison target image information storage unit 301) as comparison target image information.

  In step S27, it is determined whether or not the operation of the client application execution unit 20 by the operator 3 is completed. If the operation has not ended, the process returns to step S21. If the operation has been completed, the process proceeds to step S28.

  In step S <b> 28, the scenario generation unit 302 generates an operation scenario by combining the comparison target image information in the comparison target image information storage unit 301 and the event log information in the event log storage unit 306, and generates a scenario storage unit 112. To record.

  An example of the operation scenario in the scenario storage unit 112 is shown in FIG. This operation scenario includes, as setting items, operation steps, screen identification information (screen ID), a screen transition determination image and comparison target image of a bitmap image, determination / operation contents, and numerical information.

  In the operation scenario shown in FIG. 7, the file name of a file in which the comparison target image is saved as an appropriate image file format is set in the setting item of the comparison target image. In this example, “.bmp” representing a bitmap image format file is set.

  FIG. 9 shows the contents of the comparison target image file (2.bmp) set in [Comparison target image] in the operation scenario shown in FIG. In this example, the “travel expense clearing system” extracted from the schedule display screen 21 is registered in the operation scenario as a comparison target image.

  In the process of step S24 described above, the appearance (shape, color, etc.) of the button may change when the mouse reaches the button immediately before clicking the mouse. For example, from the state where the mouse cursor (MC) does not reach the button setting area (BA1) as shown in FIG. 15 (is located outside BA1), the mouse cursor (MC) is moved to the button setting area as shown in FIG. When (BA1) is reached (positioned within BA1), the button outline (not explicit / explicit), color (eg non-inverted / inverted), brightness (dark / light), etc. change. In fact, many application softwares on a general-purpose OS change the appearance (shape, color, etc.) just by moving the mouse over the menu.

  Therefore, if the screen image information is acquired when the mouse click occurs at the time of step S24, only the image after the change can be obtained. However, when the scenario is executed, the button image must be detected from the entire screen image in a state where the mouse is still located away from the button to be operated. The button image has not changed. That is, the time point at which the entire screen image should be acquired must be the time point after the operator 3 visually recognizes the comparison target (eg, button) and before the mouse approaches the comparison target. Must be before the subject's appearance changes. Therefore, it is necessary to operate the screen image recording instruction unit in the step S21.

  By having an image acquisition processing function for the entire screen as described above, a load test system based on an executable operation scenario can be constructed.

(Third embodiment)
Next, a third embodiment of the present invention will be described with reference to FIGS.

  The third embodiment solves the problem of the method for extracting the comparison target range at the time of scenario generation. For example, in the method in which the comparison target range extraction unit 303 in FIG. 13 extracts the screen image to identify the comparison target, the extraction range is not determined by the operator in the conventional method, and the extraction range is not freely determined. May become inappropriate.

  As a result, if a part of the extracted screen image includes a part whose image changes every time it is executed, the image for specifying the comparison target may become invalid. As a typical example in this case, there is a situation in which when a portion near the edge of a certain window is clicked, a background image (wallpaper, another window) behind the window is included.

  FIG. 17 shows an example of a screen situation in which another background window exists behind the operation target application window. In such a screen situation, an extracted image of the “meeting room reservation system” button including a part of the background image is used as a comparison target image for specifying the “meeting room reservation system” button, as shown in FIG. May be saved. FIG. 19 shows an example of a screen situation in which a background window different from that in FIG. 17 exists behind the operation target application window shown in FIG. Since the image shown in FIG. 18 does not exist in the screen situation of FIG. 19, the click operation of the “conference room reservation system” button cannot be reproduced in the screen situation of FIG. 19, and the comparison target cannot be specified. For this reason, the operation scenario including the image of FIG. 18 operates normally in the situation of FIG. 17 but does not operate normally in the situation of FIG.

  Similarly, since the extraction range is not known to the operator and the extraction range cannot be determined freely, as an example in which the extraction range becomes inappropriate, the same image as the extracted image has a plurality of operation target screens. If present at the position, the image for specifying the comparison target may become invalid.

  For example, as an image to be compared for specifying the “travel expense adjustment system” button of the application as shown in FIG. 20, an image of “expense expense adjustment” shown in FIG. 21 may be extracted. However, there are two images of “cost adjustment system” shown in FIG. 21 on the “travel expense adjustment system” button and “expense adjustment system” button, and it is not possible to specify which button is the comparison target from this image. For this reason, in the situation shown in FIG. 20, the scenario including the image of FIG. 21 may cause a problem of performing an operation different from the intention of the scenario producer.

  In the third embodiment, in order to solve the above-described problems, a mechanism for explicitly indicating the extraction range of the screen to the operator at the time of scenario generation, and the enlargement / reduction of the extraction range by the key operation at the same time as the explicit indication And a mechanism that enables deformation. FIG. 22 is an example of a screen explicitly showing the extraction range. A rectangular frame 221 in FIG. 22 indicates the extraction range, and the operator can select an appropriate extraction range by referring to the rectangular frame 221 when the mouse is clicked. FIG. 23 shows an example in which the extraction range is enlarged / reduced and modified. Here, by changing the extraction range so as to surround the entire “travel expense adjustment system” button, it is possible to prevent the occurrence of the above-described problem including the common image.

  In the third embodiment, an operation scenario describing a procedure for generating an operation event for a comparison target image including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. Is a scenario generation system (scenario generation device) for generating event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen, and indicating an extraction range in which the event information includes position coordinates At this time, based on the event information, extraction range determining means for determining extraction range information indicating a range from which the comparison target image is extracted, and the extraction range of the comparison target image based on the extraction range information According to the extraction range display means to be displayed on the screen and the enlargement or reduction instruction input to the displayed extraction range, the display is performed. Extraction range changing means for changing the extraction range information so as to expand or reduce the extraction range, and when the event information is a recording instruction following the instruction of the extraction range, on the operation screen based on the event information Image acquisition means for extracting and storing the comparison target image information from the image information of the whole image based on the coordinate value of the image information of the whole image and the extraction range information, the stored event information, and the comparison A scenario generation unit configured to generate the operation scenario based on the target image information. The event information acquisition unit is realized by an event recording unit 309 and an event log storage unit 306 shown in FIG. The extraction range determining unit and the extraction range changing unit are realized by the extraction range instruction unit 314 and the extraction range information storage unit 313. The extraction range display means is realized by the range display unit 312. The image acquisition means is realized by the screen image recording instruction unit 311, the screen image recording unit 310, the entire screen image information storage unit 307, the comparison target image information storage unit 301, and the comparison target range extraction unit 303. The scenario generation means is realized by the scenario generation unit 302 and the scenario storage unit 112.

  FIG. 24 is a schematic diagram showing a configuration of a scenario generation system according to the third exemplary embodiment of the present invention. This scenario generation system includes an extraction range instruction unit 314, an extraction range information storage unit 313, and a range display unit 312 as compared to the configuration shown in FIG.

  Next, the operation of the scenario generation system configured as described above will be described.

  First, as described above, it is assumed that the client application execution unit 20 is activated and the operation of the scenario generation system 30 is started. Similarly, the client application execution unit 20 is operated in response to various input events from the operating system execution unit (OS) 40 by a mouse or keyboard operation by the operator 3.

  The scenario generation system 30 obtains information such as various event information, image information of the client application execution unit 20, extraction range information in the image information, and position coordinates from the operating system execution unit (OS) 40. A scenario is generated and recorded in the scenario storage unit 112.

  For example, the event recording unit 309 collects event information generated in response to an operation on the operation screen and records it in the event log storage unit 306 as an event log.

  Here, when the event information is an extraction range instruction including position coordinates, the extraction range instruction unit 314 determines extraction range information indicating a range in which a comparison target image is extracted based on the event information. The range information is recorded in the extraction range information storage unit 313.

  The range display unit 312 displays a rectangular frame 231 indicating the extraction range of the comparison target image on the operation screen based on the extraction range information in the extraction range information storage unit 313.

  The operator performs key operations while looking at the rectangular frame 231. When the key operation is an enlargement or reduction instruction input to the displayed rectangular frame 231, the extraction range instruction unit 314 enlarges or displays the displayed rectangular frame 231 according to the enlargement or reduction instruction. The extraction range information in the extraction range information storage unit 313 is changed so as to be reduced.

  When the key operation is a recording instruction subsequent to the extraction range instruction, the screen image recording instruction unit 311 inputs the recording instruction to the screen image recording unit 310. The screen image recording unit 310 extracts image information of the entire image on the operation screen based on the recording instruction, and records the extracted image information in the entire screen image information storage unit 307.

  The comparison target range extraction unit 303 compares the image information of the comparison target from the operation screen based on the coordinate values of the image information of the entire image in the entire screen image information storage unit 307 and the extraction range information in the extraction range information storage unit 313. Is extracted and recorded in the comparison target image information storage unit 301.

  The scenario generation unit 112 generates an operation scenario based on the event information in the event log in the event log storage unit 306 and the image information in the comparison target image information storage unit 301 and records the operation scenario in the scenario storage unit 112.

  As described above, according to the present embodiment, the configuration in which the rectangular frame 231 indicating the extraction range of the comparison target image is displayed on the operation screen can prevent the designation of an image that causes an error when executing the scenario, and the scenario correction work Efficiency can be improved. In addition, while the rectangular frame 231 is displayed, the displayed rectangular frame 231 can be adjusted to an appropriate extraction range by the configuration in which the rectangular frame 231 is enlarged or reduced according to the input enlargement or reduction instruction. .

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described with reference to FIGS.

  In the fourth embodiment, as described in the third embodiment, when the vicinity of a certain window is clicked, the background image (wallpaper, other window) behind the window is included in the extracted image. The situation is solved.

  As described above, when the extracted image including the background image is stored as shown in FIG. 18 in order to specify the “conference room reservation system” button from the screen shown in FIG. An inconvenience that the operation of clicking the “room reservation system” button cannot be reproduced occurs.

  In the fourth embodiment, in order to eliminate such an inconvenience, a window including the operation target is specified when the operation target is operated, and the window is included in both the extraction range determined in the third embodiment and the specified window range. This is a description of the mechanism for setting the extracted range as the extraction range. However, unlike the third embodiment, the fourth embodiment may omit the display of the rectangular frame 231 that clearly indicates the extraction range. FIG. 25 shows only an operation target application window portion extracted from the extracted image shown in FIG. 18 extracted from the operation screen shown in FIG.

  The extracted image shown in FIG. 25 does not include a background image, and therefore also exists on the screen shown in FIG. From this, the operation scenario including the extracted image shown in FIG. 25 operates normally so as to specify the “conference room reservation system” on the operation screen shown in FIG. 17 or 19.

  In the fourth embodiment, an operation scenario describing a procedure for generating an operation event for a comparison target image including a partial image of the operation target in the operation screen for an application program accompanied by display of the operation screen. A scenario generation system (which is a scenario generation device that collects and stores event information that occurs in response to an operation on the operation screen, and indicates an extraction range indicating that the event information includes position coordinates. An extraction range determining means for determining extraction range information indicating a range for extracting the comparison target image based on the event information; and when the event information is a recording instruction following the extraction range instruction, the event information Based on the coordinate value of the image information of the whole image on the operation screen and the extraction range information, Image acquisition means for extracting and storing the comparison target image information, and scenario generation means for generating the operation scenario based on the stored event information and the comparison target image information. As the acquisition means, when a plurality of window screens are displayed on the operation screen, a window for storing window range information and overlapping order information indicating the range and overlapping order of the window screens based on the recording instruction, respectively. Based on the range storage means, the position coordinates included in the extraction range instruction, the window range information and the overlapping order information, among the window range information including the position coordinates, corresponds to the overlapping order information indicating the highest level. Window selection means for selecting window range information to be selected, and the window selection means Common range extraction means for extracting common range information indicating a common range between the window range information and the extraction range information, and an extraction range change for changing the value of the extraction range information in the storage means to the value of the common range information And an image extraction storage means for extracting and storing the image information to be compared based on the changed extraction range information of the value. Instead of the extraction range changing means, based on the position coordinates included in the extraction range instruction, the window range information and the overlapping order information, window range information that does not include the position coordinates and includes a part of the extraction range A background window selection means for selecting background window range information corresponding to the overlapping order information indicating the highest level, and the background window including the common range indicated by the common range information. Extraction range moving means for extracting extraction range information indicating the extraction range moved so as not to be included in the window screen range indicated by the window range information, and changing the determined extraction range information to the extracted extraction range information And a changing unit that performs the above-described processing. The event information acquisition means is realized by an event recording unit 309 and an event log storage unit 306 shown in FIG. The extraction range determining unit and the extraction range changing unit are realized by the extraction range instruction unit 314 and the extraction range information storage unit 313. The extraction range display means is realized by the range display unit 312. The image acquisition means is realized by the screen image recording instruction unit 311, the screen image recording unit 310, the entire screen image information storage unit 307, the comparison target image information storage unit 301, and the comparison target range extraction unit 303. The window range storage unit is realized by the screen image recording instruction unit 311, the screen image recording unit 310, the window range information storage unit 316, and the window range detection unit 317. Window selection means, background window selection means, common range extraction means, extraction range change means, change means and image extraction storage means are realized by comparison target range extraction section 303, comparison target image information storage section 301 and common range extraction section 315. Is done. The scenario generation means is realized by the scenario generation unit 302 and the scenario storage unit 112.

  FIG. 26 is a schematic diagram showing a configuration of a scenario generation system according to the fourth embodiment of the present invention. In this scenario generation system, a common range extraction unit 315, a window range information storage unit 316, and a window range detection unit 317 are added as compared with the configuration shown in FIG.

  In the window range information storage unit 316, as shown in an example in FIG. 27, window range information (upper left corner) is associated with the entire screen image information in the entire screen image information storage unit 307 for each screen ID of each window screen. Coordinates, lower right corner coordinates) and overlap order information are stored. The window range information is information indicating the range of each window screen. Here, the position coordinates of the upper left corner and the position coordinates of the lower right corner of the window screen are used. The overlapping order information is information indicating the overlapping order between the window screens, and in this example, the upper order (front side, front side of the screen) is shown in ascending order. The window range information and the overlapping order information are acquired by the window range detection unit 317 from an operating system (OS) execution unit 40 (not shown).

  Next, the operation of the scenario generation system configured as described above will be described.

  First, as described above, it is assumed that the client application execution unit 20 is activated and the operation of the scenario generation system 30 is started. Similarly, the client application execution unit 20 is operated in response to various input events from the operating system execution unit (OS) 40 by a mouse or keyboard operation by the operator 3.

  The scenario generation system 30 obtains information such as various event information, image information of the client application execution unit 20, extraction range information in the image information, and position coordinates from the operating system execution unit (OS) 40. A scenario is generated and recorded in the scenario storage unit 112.

  For example, the event recording unit 309 collects event information generated in response to an operation on the operation screen and records it in the event log storage unit 306 as an event log.

  Here, when the event information is an extraction range instruction including position coordinates, the extraction range instruction unit 314 determines extraction range information indicating a range in which a comparison target image is extracted based on the event information. The range information is recorded in the extraction range information storage unit 313.

  The range display unit 312 displays a rectangular frame indicating the extraction range of the comparison target image on the operation screen based on the extraction range information in the extraction range information storage unit 313. However, the display of the rectangular frame by the range display unit 312 may be omitted.

  When the event information is a recording instruction following the extraction range instruction, the screen image recording instruction unit 311 inputs the recording instruction to the screen image recording unit 310. The screen image recording unit 310 extracts image information of the entire image on the operation screen based on the recording instruction, and records the extracted image information in the entire screen image information storage unit 307.

  Here, as shown in FIG. 28, when a plurality of window screens W1 to W3 are displayed on the operation screen, the window range detection unit 317 determines the range of each window screen and the overlapping order based on the recording instruction. Are recorded in the window range information storage unit 316.

  Based on the position coordinates (xc, yc) included in the instruction of the extraction range C, the window range information in the window range information storage unit 316, and the overlapping order information, the common range extraction unit 315 performs the position coordinates (xc, Among the window range information including yc) (corresponding to W2), the window range information (corresponding to W2) corresponding to the overlapping order information indicating the highest order is selected.

  Subsequently, as shown in FIG. 29, the common range extraction unit 315 includes common range information (common range information) indicating the common range of the selected window range information (corresponding to W2) and the extraction range information (xc1, yc1, xc2, yc2). xc1 ′, yc1 ′, xc2, yc2) are extracted.

  Further, the common range extraction unit 315 changes the value of the extraction range information in the extraction range information storage unit 313 to the value of the common range information. Note that, instead of the method shown in FIG. 29, the common range extraction unit 315 performs the position detection based on the position coordinates included in the extraction range instruction, the window range information, and the overlapping order information, as shown in FIG. 30. While selecting the background window range information (corresponding to W1) corresponding to the overlapping order information indicating the highest rank from among the window range information not including coordinates and including a part of the extraction range, including the common range indicated by the common range information The extraction range information indicating the extraction range moved so as not to be included in the range of the window screen W1 indicated by the background window range information is extracted, and the extraction range information in the extraction range information storage unit 313 is extracted as the extraction range after the movement. You may operate | move so that it may change into information (xc1 ', yc1', xc2 ', yc2').

  In any case, after changing the extraction range information, the comparison target range extraction unit 303 includes the coordinate value of the image information of the entire image in the entire screen image information storage unit 307 and the extraction range information in the extraction range information storage unit 313. The image information to be compared is extracted from the operation screen based on the above, and is recorded in the comparison image information storage unit 301.

  The scenario generation unit 112 generates an operation scenario based on the event information in the event log in the event log storage unit 306 and the image information in the comparison target image information storage unit 301 and records the operation scenario in the scenario storage unit 112.

  As described above, according to the present embodiment, the configuration for extracting the comparison target image based on the common range of the uppermost window range information including the position coordinates included in the extraction range instruction and the extraction range information, Since the background window screen that does not include the designated position coordinates in the extraction range is not extracted as part of the comparison target image, it is possible to prevent the designation of an image that causes an error during scenario execution, and to improve the efficiency of scenario correction.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described with reference to FIGS.

  In the second, third, and fourth embodiments, there is an error in the generated scenario, and even when an error occurs during execution, the error does not become apparent until the scenario is executed.

  For example, a situation is assumed in which a scenario for the screen shown in FIG. 6 is generated by the method shown in the second embodiment.

  It is assumed that the comparison target image information of FIG. 31 is obtained by an operation of clicking “Expense Clearing System” on the screen of FIG. 6, and a scenario including the information is generated and recorded in the scenario storage unit 112.

  When the operation scenario in the scenario storage unit 112 is executed, the above comparison target image information matches not only the “expense clearing system” button intended by the operator but also the “travel expense clearing system” button. End up. Therefore, in some situations, a malfunction may occur in which the system clicks on the “travel expense clearing system”.

  This is a problem when the method of selecting a comparison target range is inappropriate in the scenario generation process, and there are a plurality of portions where the comparison target image information matches on the operation screen. This problem is not discovered until a malfunction occurs after the scenario is actually operated. After the malfunction occurs, the operator needs to investigate the cause of the malfunction, understand that the method of selecting the comparison target range is inappropriate, and then perform the work of correcting a part of the scenario again. This increases the turnaround time from scenario creation to modification. When there are a plurality of such errors in the same scenario, it takes a long time to correct all the errors and obtain a complete scenario. Furthermore, there is a possibility that the malfunction as described above may cause a malfunction that is difficult for the operator to find. For example, even if the “travel expense clearing system” is clicked instead of the “expense clearing system”, the subsequent screen transitions are very similar in both cases, and it may be difficult for the operator to immediately find a malfunction. In such a case, it takes time and effort for the operator to recognize that the comparison target image has a cause of malfunction.

  In the fifth embodiment, in order to shorten the time from scenario creation to correction, when there is an error in the scenario, the error is pointed out to the operator before the scenario is executed, and the opportunity for correction The “scenario pre-check / correction system” is realized.

  FIG. 32 is a schematic diagram showing a scenario preliminary check / correction system and its peripheral configuration according to the fifth embodiment of the present invention.

  The scenario preliminary check / correction system 50 checks and corrects the operation scenario before executing the operation scenario in the scenario storage unit 112 generated by the scenario generation system 30. The scenario pre-check / correction system 50 has a function of displaying an error content to an operator and giving an opportunity for correction if an error is found in the scenario. A configuration including both the scenario pre-check / correction system 50 and the scenario generation system 30 may be referred to as a scenario generation device.

  FIG. 33 is a block diagram showing the relationship between the scenario generation system 30 and the scenario pre-check / correction system 50.

  The scenario preliminary check / correction system 50 includes a scenario preliminary check unit 501 and a scenario preliminary correction unit 502.

  The scenario generation system 30 differs from that shown in FIG. 13 in the following points.

  When generating an operation scenario, the scenario generation unit 302 takes in the entire screen image information in the entire screen image information storage unit 307 and adds it to the scenario. That is, the scenario generation unit 302 operates based on the comparison target image information in the comparison target image information storage unit 301, the event log in the event log storage unit 306, and the entire screen image information in the entire screen image information storage unit 307. A scenario is generated and recorded in the scenario storage unit 112-2. Accordingly, the scenario storage unit 112-2, for a certain step number, event information generated by an operation on the operation screen, image information of the entire image on the operation screen, and a comparison target extracted from the operation screen The operation scenarios are associated with each other. Here, FIG. 34 shows an example of the contents of the operation scenario in the scenario storage unit 112-2. The difference from FIG. 7 is that an item for the entire screen image is added.

  As shown in FIG. 35, the scenario preliminary check unit 501 includes a scenario step information extraction unit 5011, a matching determination unit 5014, and a multiple image warning unit 5015.

  Here, the scenario step information extraction unit 5011 extracts the image information 5012 of the entire image for a certain step number from the operation scenario in the scenario storage unit 112-2 and sends it to the matching determination unit 5014, and the scenario. It has a function of extracting image information 5013 to be compared with respect to a certain step number from the operation scenario in the storage unit 112-2 and sending it to the matching determination unit 5017.

  The matching determination unit 5014 determines whether or not the comparison target image information extracted by the scenario step information extraction unit 5011 can be uniquely determined from the image information of the entire image extracted by the scenario step information extraction unit 5011. If the result of determination is negative, the multi-image warning unit 5015 for displaying the negative is provided.

  The multiple image warning unit 5015 has a function of displaying a warning screen when activated by the matching determination unit 5014.

  As shown in FIG. 36, the scenario pre-correction unit 502 includes an entire screen image extraction unit 5021, an entire screen image display unit 5022, a comparison target image range instruction unit 5023, a comparison target image range extraction unit 5024, and a comparison target image storage unit 5025. It has.

  Here, the entire screen image extraction unit 5021 has a function of extracting image information of the entire image from the operation scenario for the step number on which the warning screen is displayed.

  The entire screen image display unit 5022 has a function of displaying the entire image extracted by the entire screen image extraction unit 5021.

  The comparison target image range instructing unit 5023 receives a reextraction range instruction in which two position coordinates are set to the vertices of diagonal lines or a recording instruction following the reextraction range instruction by the operation of the operator. It has a function to input to.

  When the whole image is displayed by the whole image extraction display unit 5022, the comparison target image range extraction unit 5024 displays a display based on the event information when the event information indicates a re-extraction range having two position coordinates as diagonal vertices. A function for extracting extraction range information indicating a range for extracting the image to be compared from the entire image in the image, and when the event information is a recording instruction following the instruction of the re-extraction range, the entire displayed image is displayed based on the event information. A function for extracting image information for comparison from image information of the entire image based on the coordinate value of the image information of the image and the extracted extraction range information, and a comparison target image storage unit for storing the extracted image information for comparison The comparison target image information in the operation scenario in which the image information of the entire image is displayed by the function for writing to the image 5025 and the entire screen image display unit 5022 And a function of overwriting the image information to be compared in 憶部 5025.

  The comparison target image storage unit 5025 is a storage device that can be read / written from the comparison target image range extraction unit 5024, and temporarily stores image information to be compared.

  Next, operations of the scenario generation system and the scenario pre-check / correction system configured as described above will be described.

  In the processing flow shown in FIG. 38, in step S31, the scenario generation system 50 is activated by the operation of the operator, generates an operation scenario, and records it in the scenario storage unit 112-2.

  In step S32, the scenario preliminary check / correction system 50 is activated by the operation of the operator.

  In step S33, the scenario pre-check unit 501 determines a determination target step in the scenario. Further, the scenario step information extraction unit 5011 sequentially selects each step of the operation scenario in the scenario storage unit 112-2 and extracts information included in the step. Then, the entire screen image information 5012 and the comparison target image information 5013 are extracted from them and sent to the matching determination unit 5014.

  In step S34, the matching determination unit 5014 determines whether the comparison target image information 5013 can be uniquely determined in the entire screen image information 5012. Here, the fact that it can be determined uniquely means that the portion of the entire screen image information 5012 that matches the comparison target image information 5013 is unique. For example, in the state where the schedule display screen 21 shown in FIG. 4 is the entire screen image information 5012, the image extracted as a comparison target is the “settlement system” shown in FIG. Then, it is determined that the image extracted as the comparison target cannot be uniquely determined as the “travel expense settlement system” or the “expense settlement system”.

  The matching determination unit 5014 extracts all portions including an image having the same size as the comparison target image information from the entire screen image information 5012, and determines whether the comparison target image information matches the color information of all pixels. Can be inspected. Note that “match” here refers to the same standard as the image comparison unit 133 described in the first embodiment.

  If the matching determination unit 5014 determines that it cannot be uniquely determined, the multi-image warning unit 5015 is activated. The multi-image warning unit 5015 uses the warning screen to point out that the operator has not been able to uniquely determine the image, and displays the entire screen image information 5012, which matches the comparison target image information 5013. Whether there is an image is displayed in the comparison target image information 5012 to let the operator know the location. For example, a display method is conceivable in which a frame having the same size as the outline of the comparison target image information 5013 is displayed in the entire screen image information 5012 where the comparison target image information 5013 matches.

  In step S35, when the matching determination unit 5014 determines that it cannot be uniquely determined, the scenario pre-correction unit 502 is automatically activated to provide a user interface for the operator to correct the scenario.

  When activated, the scenario pre-correction unit 502 corrects the corresponding step according to the operation of the operator. Specifically, the entire screen image extraction unit 5021 extracts the entire screen image information 5012 included in the step to be corrected from the operation scenario in the scenario storage unit 112-2 and sends it to the entire screen image display unit 5022. To do.

  The entire screen image display unit 5022 displays the entire screen image information 5012 as an image in a form that can be visually recognized by the operator.

  The comparison target image range instruction unit 5023 provides an environment in which the operator can specify the comparison target image range using the mouse on the displayed image of the entire screen image information 501. That is, on the displayed image of the entire screen image information 501, the operator performs drag and drop with the mouse, for example. A rectangle in which the coordinates at which the mouse drag is started and the coordinates at which the drop is performed are used as vertices that form a diagonal line is used as an instruction for the comparison target image range.

  The comparison target image range instruction unit 5023 sends the coordinate information related to the comparison target image range specified by the operator as described above to the comparison target image range extraction unit 5024.

  The comparison target image range extraction unit 5024 extracts image information of a portion included in the comparison target image range from the entire screen image, and writes it in the comparison target image storage unit 5025. Thereafter, the comparison target image range extraction unit 5024 applies the comparison target image information in the comparison target image storage unit 5025 to the information corresponding to the comparison target image of the corresponding step of the operation scenario in the scenario storage unit 112-2. And overwrite.

  For example, in a situation where it is pointed out that the image in FIG. 31 is not uniquely determined, the operator designates a comparative image range that is wider in the horizontal direction with the mouse in the comparative image range instruction unit 502. Thus, a comparison target image as shown in FIG. 37 is obtained.

  When the correction work in the scenario pre-correction unit 502 is completed, the scenario pre-check unit 501 is automatically activated and the process returns to step S34, and the step of the operation scenario corrected above uniquely identifies the comparison target image. Judge whether or not.

  Next, in step S36, the scenario preliminary check unit 501 determines whether or not the determination of all steps of the operation scenario in the scenario storage unit 112-2 has been completed, and if completed, ends the processing. If it has not been completed, the scenario pre-check unit 501 proceeds to step S37.

  In step S37, the scenario preliminary checking unit 501 updates the operation scenario determination target step in the scenario storage unit 112-2 by +1 row, and returns to step S33.

  With the above operation, the operator can correct the scenario error with less effort before executing the operation scenario.

  As described above, according to the present embodiment, it is determined whether or not the extracted image information to be compared can be uniquely determined. If not, an image that causes an error during scenario execution is configured by displaying a warning screen. Designation can be prevented, and the work efficiency of scenario correction can be improved.

  Note that the present embodiment is not limited to the procedure shown in FIG. 38, and for example, as shown in FIG. 39, after checking all the steps, the steps corresponding to the error may be modified. That is, instead of step S35, in step S35a, the corresponding step is displayed in red, and the process proceeds to step S36. Note that step S35a is not limited to display in red, but can be applied to any display that can distinguish the corresponding step, such as an arbitrary color, mark, or blinking display. Further, in step S35a, instead of changing the display, the number of the step corresponding to the error or information (title) that can be identified with it may be displayed as a list. On the other hand, if it is determined in step S36 that all steps have been completed, the process proceeds to step S38. In step S38, the scenario pre-correction unit 502 is automatically activated to provide a user interface for the operator to correct the scenario for the corresponding step displayed in red in step S35a. In step S38, when the operation scenarios of all corresponding steps are corrected, the scenario pre-correction unit 502 ends the process.

  Further, in the present embodiment, not all the steps to be checked may be checked, but only one or a plurality of steps may be targeted.

  The method for selecting the steps to be targeted is not particularly limited. For example, the operator may explicitly specify the target step, or the scenario pre-check / correction system 50 sequentially checks the steps in the scenario until a specific number of errors are found. It is good also as a set.

Further, each of the above embodiments is not limited to the case where it is applied to a load test. For example, it can be applied to any application for automatically operating an application having a GUI function, such as a function test, monitoring, or information collection. The effect of this can be obtained.
For example, the case where it applies to a functional test is described. In the process of software development, when the software to be developed has a GUI function, a so-called function test is performed to verify the behavior of the software with respect to the GUI operation. Each embodiment can be applied to the use of a tool for operating a GUI on behalf of an operator during the function test. In addition, abnormal operation may occur due to software bugs during functional testing. When this operation abnormality occurs due to delicate timing between processing in the software and GUI operation, it is necessary to reproduce the operation abnormality by repeatedly executing a predetermined GUI operation. Each embodiment can be similarly applied to the repeated execution of the GUI operation during such a function test.

  Next, the case of applying to monitoring will be described. The remote WWW (world wide web) browser GUI is periodically operated to access the WWW server to be monitored, and by checking the access result, it is periodically determined whether the WWW server is operating normally. You may want to monitor. In such monitoring, each embodiment can be applied to a use in which the GUI function of the WWW browser is automatically operated periodically instead of the operator.

  Similarly, a case where it is applied to information collection will be described. In some cases, a remote WWW server periodically accesses the GUI of the WWW browser and periodically collects information from the WWW browser. When collecting such information, each embodiment can be applied to a purpose of automatically operating the GUI of the WWW browser on behalf of the operator on a regular basis.

  In addition to the applications described above, the above embodiments can be applied to all applications in which an application having a GUI function is operated on behalf of the operator.

  In addition, the method described in the above embodiment includes a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a magneto-optical disk (MO) as programs that can be executed by a computer. ), And can be distributed in a storage medium such as a semiconductor memory.

  In addition, as long as the storage medium can store a program and can be read by a computer, the storage format may be any form.

  In addition, an OS (operating system) running on a computer based on an instruction of a program installed in the computer from a storage medium, MW (middleware) such as database management software, network software, and the like realize the above-described embodiment. A part of each process may be executed.

  Further, the storage medium in the present invention is not limited to a medium independent of a computer, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the number of storage media is not limited to one, and the case where the processing in the above embodiment is executed from a plurality of media is also included in the storage media in the present invention, and the media configuration may be any configuration.

  The computer according to the present invention executes each process in the above-described embodiment based on a program stored in a storage medium, and is a single device such as a personal computer or a system in which a plurality of devices are connected to a network. Any configuration may be used.

  In addition, the computer in the present invention is not limited to a personal computer, but includes an arithmetic processing device, a microcomputer, and the like included in an information processing device, and is a generic term for devices and devices that can realize the functions of the present invention by a program. .

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram illustrating a configuration of an entire system that performs a load test according to a first embodiment of the present invention. It is a block diagram which shows the structure of the load generation system which concerns on the said 1st Embodiment. It is a block diagram which shows the structure of the comparison object specific | specification part which concerns on the said 1st Embodiment. It is a figure which shows the specific example of the image information which the client application based on the said 1st Embodiment is displaying on the screen. It is a figure which shows the hierarchy example of the schedule display screen which concerns on the said 1st Embodiment. It is a figure which shows the coordinate axis of the schedule display screen which concerns on the said 1st Embodiment. It is a figure which shows the example of a partial description of the operation scenario which concerns on the said 1st Embodiment. It is a figure which shows the specific example of the screen transition determination image which concerns on the said 1st Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the said 1st Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the said 1st Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the said 1st Embodiment. It is a block diagram which shows the structure of the whole system which produces the scenario based on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the scenario generation system which concerns on the said 2nd Embodiment. It is a flowchart which shows the procedure of the process which concerns on the said 2nd Embodiment. It is a figure for demonstrating the characteristic of the process which concerns on the said 2nd Embodiment. It is a figure for demonstrating the characteristic of the process which concerns on the said 2nd Embodiment. It is a figure which shows the specific example of the several window screen which concerns on the 3rd Embodiment of this invention. It is a figure which shows the specific example of the comparison target image which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the several window screen which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the schedule display screen which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the extraction range which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the extraction range which concerns on the said 3rd Embodiment. It is a block diagram which shows the structure of the scenario generation system which concerns on the said 3rd Embodiment. It is a figure which shows the specific example of the extraction range which concerns on the said 3rd Embodiment. It is a block diagram which shows the structure of the scenario generation system which concerns on the 4th Embodiment of this invention. It is a figure which shows the specific example of the whole screen image information, window range information, and overlapping range information which concern on the said 4th Embodiment. It is a figure which shows the specific example of the whole screen image information, window range information, and overlapping range information which concern on the said 4th Embodiment. It is a figure which shows the specific example of the whole screen image information, window range information, and overlapping range information which concern on the said 4th Embodiment. It is a figure which shows the specific example of the whole screen image information, window range information, and overlapping range information which concern on the said 4th Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the 5th Embodiment of this invention. It is a schematic diagram which shows the scenario prior check / correction system which concerns on the said 5th Embodiment, and its periphery structure. It is a block diagram which shows the relationship between the scenario production | generation system which concerns on the said 5th Embodiment, and a scenario prior check / correction system. It is a figure which shows the example of a part description of the operation scenario which concerns on the said 5th Embodiment. It is a block diagram which shows the structure of the scenario prior check part which concerns on the said 5th Embodiment. It is a block diagram which shows the structure of the scenario prior correction part which concerns on the said 5th Embodiment. It is a figure which shows the specific example of the comparison target image which concerns on the said 5th Embodiment. It is a flowchart which shows the procedure of the process which concerns on the said 5th Embodiment. It is a flowchart which shows the modification of the procedure of the process which concerns on the said 5th Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Information terminal (PC), 2 ... Server, 3 ... Operator, 10 ... Load generation system, 11 control part, 12 ... Screen transition determination part, 13 ... Comparison target specific part, 14 ... Operation event generation part, 15 ... Image information acquisition unit, 20 ... client application execution unit, 30 ... scenario generation system, 40 ... operating system (OS) execution unit, 111 ... operation execution control unit, 112 ... operation scenario, 131 ... comparison target specifying control unit, 132 ... Comparison target determination unit, 133 ... Image comparison unit, 134 ... Relative positional relationship analysis unit, 302 ... Scenario generation unit, 303 ... Comparison target range extraction unit, 304 ... Shape / size change unit, 305 ... Matching determination unit, 308 ... Event Detection unit, 309 ... event recording unit, 310 ... screen image recording unit, 311 ... screen image recording instruction unit.

Claims (9)

The target application program with the display of the operation screen, the operation screen in the operation target image obtained by cutting out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A scenario generation device that generates an operation scenario describing a procedure for generating an operation event,
Event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen;
When the operation on the operation screen is an operation for generating the operation scenario different from the operation for the application program and the event information is an instruction to determine an extraction range including position coordinates, based on the event information, Extraction range determination means for determining extraction range information indicating the range of extracting the image to be compared with the coordinates of both vertices forming a diagonal line of a predetermined rectangular frame centered on the position coordinate ;
Based on the extraction range information, an extraction range display means for displaying the rectangular frame indicating the extraction range of the comparison target image on the operation screen;
When the operation on the operation screen is an operation for generating the operation scenario and the event information is a recording instruction following the instruction to determine the extraction range, an entire image on the operation screen is based on the event information. Image acquisition for extracting and storing the comparison target image information from the image information of the entire image based on the coordinate values within the rectangular frame indicated by the extraction range information among the coordinate values of the image information Means,
A scenario generation unit configured to generate the operation scenario based on event information generated when an operation on the operation screen is performed on the application program among the stored event information and the image information to be compared; A scenario generation device characterized by that. The scenario generation device according to claim 1,
The apparatus further comprises extraction range changing means for changing the extraction range information so as to enlarge or reduce the displayed extraction range in response to an enlargement or reduction instruction input to the displayed extraction range. A feature scenario generator. The target application program with the display of the operation screen, the operation screen in the operation target image obtained by cutting out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A scenario generation device that generates an operation scenario describing a procedure for generating an operation event,
Event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen;
When the operation on the operation screen is an operation for generating the operation scenario different from the operation for the application program and the event information is an instruction to determine an extraction range including position coordinates, based on the event information, Extraction range determination means for determining extraction range information indicating the range of extracting the image to be compared with the coordinates of both vertices forming a diagonal line of a predetermined rectangular frame centered on the position coordinate ;
When the operation on the operation screen is an operation for generating the operation scenario and the event information is a recording instruction following the instruction of the extraction range, the entire image on the operation screen is based on the event information. Image acquisition means for extracting and storing the comparison target image information from the image information of the whole image based on the coordinate value within the rectangular frame indicated by the extraction range information among the coordinate values of the image information When,
A scenario generation unit configured to generate the operation scenario based on event information generated when an operation on the operation screen is performed on the application program among the stored event information and the image information to be compared; And
The image acquisition means includes
When a plurality of window screens are displayed on the operation screen, based on the recording instruction, window range storage means for storing window range information and overlap order information indicating the range of each window screen and the overlap order, respectively,
Based on the position coordinates included in the extraction range determination instruction, the window range information, and the overlapping order information, the window range information corresponding to the overlapping order information indicating the highest rank among the window range information including the position coordinates. A window selection means for selecting
Common range extraction means for extracting common range information indicating a common range between the window range information selected by the window selection means and the extraction range information;
Extraction range changing means for changing the value of the extraction range information to the value of the common range information;
Image extraction storage means for extracting and storing the image information to be compared based on the extraction range information of the changed value;
A scenario generation device comprising: The scenario generation device according to claim 3,
Instead of the extraction range changing means,
Based on the position coordinates included in the extraction range determination instruction, the window range information, and the overlapping order information, the highest position is indicated among the window range information not including the position coordinates and including a part of the extraction range. Background window selection means for selecting background window range information corresponding to the overlapping order information;
Extraction range moving means for extracting extraction range information indicating an extraction range that is moved so as not to be included in the range of the window screen indicated by the background window range information while including the common range indicated by the common range information;
Changing means for changing the determined extraction range information to the extracted extraction range information;
A scenario generation device comprising: The target application program with the display of the operation screen, the operation screen in the operation target image to cut out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A scenario generation device that generates an operation scenario describing a procedure for generating an operation event,
For a certain step number, event information generated with the operation on the operation screen, comprising in association image information of the entire image on the operation screen, and the image information, the comparison extracted from the operation screen each other Storage means for storing the operation scenario;
Image extraction means for extracting the image information of the entire image and the image information of the comparison object for a certain step number from the operation scenario in the storage means;
In the image information of the entire image extracted by the image extracting means, determining means for determining the image information matching part one place Dakea Luke not to be compared, which is extracted by the image extracting unit,
If the result of the determination is NO, a result display means for displaying that the result is NO,
A scenario generation device comprising: The scenario generation device according to claim 5,
Whole image extraction and display means for extracting the image information of the whole image from the operation scenario and displaying the whole image for the step number displayed as the negative,
While displaying the whole image by the whole image extraction and display means, event information generated by the operation for generating the operation scenario different from the operation on the application program on the whole screen is at two positions. when extraction instruction of re-extraction range in which the coordinates in the diagonal of the apex, compared for extracting extraction range information shown on the basis of the event information, a range for extracting the image to be compared from the entire screen in the two coordinates Image range extraction means;
When the operation on the operation screen is an operation for generating the operation scenario and the event information is a recording instruction following the instruction of the re-extraction range, an image of the whole image being displayed based on the event information Among the coordinate values of the information , based on the coordinate values included in the re-extraction range in which the two position coordinates indicated by the extracted extraction range information are the vertices of diagonal lines, the image information of the entire image is compared. Comparison target image extraction means for extracting image information;
Image updating means for overwriting and updating the comparison target image information in the operation scenario in which the image information of the whole image is displayed by the whole image extraction display means with the comparison target image information extracted by the comparison target image extraction means;
A scenario generation device comprising: The target application program with the display of the operation screen, the operation screen in the operation target image obtained by cutting out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A program used for a scenario generation apparatus that generates an operation scenario describing a procedure for generating an operation event and includes a storage means,
The scenario generation device;
Event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen;
When the operation on the operation screen is an operation for generating the operation scenario different from the operation for the application program and the event information is an instruction to determine an extraction range including position coordinates, based on the event information, Extraction range determination means for determining extraction range information indicating the range of extracting the image to be compared with the coordinates of both vertices forming a diagonal line of a predetermined rectangular frame centered on the position coordinate ;
Extraction range display means for displaying the rectangular frame indicating the extraction range of the comparison target image on the operation screen based on the extraction range information;
When the operation on the operation screen is an operation for generating the operation scenario and the event information is a recording instruction following the instruction of the extraction range, the entire image on the operation screen is based on the event information. Of the coordinate values of the image information, the image information to be compared is extracted from the image information of the whole image based on the coordinate values within the range of the rectangular frame indicated by the extraction range information , and is written in the storage means. Image acquisition means,
Of the stored event information, scenario generation means for generating the operation scenario based on event information generated when an operation on the operation screen is performed on the application program, and the image information to be compared,
Program to function as. The target application program with the display of the operation screen, the operation screen in the operation target image obtained by cutting out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A program used for a scenario generation apparatus that generates an operation scenario describing a procedure for generating an operation event and includes a storage means,
The scenario generation device;
Event information acquisition means for collecting and storing event information generated in response to an operation on the operation screen;
When the operation on the operation screen is an operation for generating the operation scenario different from the operation for the application program and the event information is an instruction to determine an extraction range including position coordinates, based on the event information, Extraction range determination means for determining extraction range information indicating the range of extracting the image to be compared with the coordinates of both vertices forming a diagonal line of a predetermined rectangular frame centered on the position coordinate ;
When the operation on the operation screen is an operation for generating the operation scenario and the event information is a recording instruction following the instruction of the extraction range, the entire image on the operation screen is based on the event information. Of the coordinate values of the image information, the image information to be compared is extracted from the image information of the whole image based on the coordinate values within the range of the rectangular frame indicated by the extraction range information , and is written in the storage means. Image acquisition means,
Of the stored event information, scenario generation means for generating the operation scenario based on event information generated when an operation on the operation screen is performed on the application program, and the image information to be compared,
Function as
The image acquisition means includes
When a plurality of window screens are displayed on the operation screen, based on the recording instruction, window range storage means for storing window range information and overlap order information indicating the range of each window screen and the overlap order, respectively,
Based on the position coordinates included in the extraction range determination instruction, the window range information, and the overlapping order information, the window range information corresponding to the overlapping order information indicating the highest rank among the window range information including the position coordinates. A window selection means for selecting
Common range extraction means for extracting common range information indicating a common range between the window range information selected by the window selection means and the extraction range information;
Extraction range changing means for changing the value of the extraction range information to the value of the common range information;
Image extraction storage means for extracting and storing the image information to be compared based on the extraction range information of the changed value;
Including programs. The target application program with the display of the operation screen, the operation screen in the operation target image to cut out a part component of the image only image pairs 応付 comparison object to be compared to the operation screen Nde free A program used for a scenario generation apparatus that generates an operation scenario describing a procedure for generating an operation event and includes a storage means,
The scenario generation device;
For a certain step number, event information generated with the operation on the operation screen, comprising in association image information of the entire image on the operation screen, and the image information, the comparison extracted from the operation screen each other Means for recording the operation scenario in the storage means;
Image extraction means for extracting the image information of the entire image and the image information of the comparison object for each step number from the operation scenario in the storage means;
The image in the image information of the entire image extracted by the extracting means, determining means whether the image information matching part one place Dakea Luke not to be compared, which is extracted by the image extracting unit,
If the result of the determination is NO, a result display means for displaying that the result is NO,
Program to function as.

Images