JP2008225683A - Screen operation system and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a screen operation system allowing specification of an individual object by designating relative positional relation of a plurality of objects even in a situation where the operation target object cannot be uniquely determined by only designating a shape of the individual object in specification of the object made to an operation target. <P>SOLUTION: When a reference image present inside an operation screen is defined in an operation scenario 112 and when a distance is defined as relative position information, a target object specification part 13 acquires distance information from the reference image about each object of the same shape as the object made to the operation target present inside the operation screen, and specifies the object made to the operation target based on the distance information. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a screen operation system and a program suitable for being applied to a load test and a function test of a computer system.

  When a large amount of processing requests are generated for the information system, the load on the information system increases. In this situation, the performance of the information system is extremely important. In other words, how much load an information system can withstand, and how good the response time is when a large amount of load is generated, is an important issue for users of the information system. is there.

  When developing and operating an information system, a load test must be performed to measure and verify the performance of the information system in order to measure and verify the characteristics at high loads. There are many. In this type of load test, as shown in Patent Document 1, it is common to take a technique of generating a large amount of processing requests to an information system in a pseudo manner from a machine corresponding to an information terminal. A machine corresponding to this information terminal often generates a load by installing a load test program. This load test program generates the same processing request from the information terminal as when a human operates the information terminal.

  There are several ways to implement a load test program. Among them, a relatively widespread method is to record a message sent from the client software to the server and generate the recorded message from the load generation tool to the server repeatedly in a pseudo manner. . However, in this system, in the case of a complicated system in which the message contents change every time, it is necessary to implement a mechanism for changing the message contents in the load generation tool in the same way as the client software. In some situations, this can be very complex and difficult and requires a lot of effort. In addition, in order to understand how the message contents change, it may be necessary to understand the internal behavior of the client software. Because it is required, there is a problem in securing human resources for load testing.

  On the other hand, as another realization method, a method of generating a load by automatic operation of a screen of client software can be considered. This is because the client software is activated in the information terminal, and the client software is operated by the load test tool performing a pseudo operation similar to that performed by humans on the client software. As a result, a processing request is generated from the client software to the server of the information system. At this time, an input event (operation event) from a mouse or a keyboard is input to a predetermined operation target such as a button in accordance with the screen transition of the client software. The contents of this series of pseudo action executions are defined in the form of operation scenarios in advance. In this method, since the mechanism implemented in the client software is used as it is to generate a message to be sent to the server, it is not necessary to create a separate mechanism for changing the message contents as described above. . Furthermore, anyone who understands how to operate the client software can create the above operation scenario and does not require advanced skills.

  In this load generation method by automatic screen operation, an object such as a button or a text box displayed in a window on the information terminal is set as an operation target. When a predetermined operation is input to this object, several realization methods can be considered. One way is to understand the structure of the software that makes up the screen and clarify the mechanism by which the software manages the objects on the screen, so that the software can identify the objects on the screen. The ID being used is clarified, and the ID and the type of event to be input there are described in the operation scenario. At the time of execution, a mouse click event or the like is input to the object based on the object ID. However, in this method, when the software object management method changes, the object specification method and the display position specification method also change, and the load test program can be changed according to the individual software object management method. There is a problem that a great deal of cost is generated because it must be developed.

  On the other hand, by comparing the object on the screen with the image information acquired in advance, the display position of the object on the screen is specified, and the mouse can be moved to the display position of the object to perform operations such as clicking. A screen operation method in which a corresponding event is generated can be considered. This method has an advantage that it is not necessary to prepare a corresponding method individually corresponding to the object management method of the window in order to specify the object based on the appearance (shape) of the object.

As an example of software that performs a pseudo operation on a software screen in such a manner, there is AutoMouse published as free software as shown in Non-Patent Document 1.
JP 2003-208329 A AutoMouse (http://ryujin.kuis.kyoto-u.ac.jp/ylab/yamakaku/AutoMouse/index_J.html)

  However, there are at least two problems in using the above-described screen operation method for client software having various screens and using them efficiently. One problem is that if there are multiple objects with the same appearance on the same screen in order to identify the object with the image information as its appearance (shape), the desired one is identified. It is impossible. Another problem is the amount of effort for creating the operation scenario. In the conventional method, image information as a snapshot of the screen of the information terminal (PC) is acquired, and a part of the image to be operated is designated from among them using a special editor. It was also necessary to define what input events should be input from the mouse and keyboard. For client software of a type that sequentially switches a plurality of operation screens, it takes a lot of labor to create an operation scenario in such a procedure, which has been a factor of remarkably reducing work efficiency.

  The present invention has been made in view of the above circumstances, and even when there are a plurality of objects having the same shape as the object to be operated in the same operation screen, the object to be operated can be specified, thereby the client application. An object of the present invention is to provide a screen operation system that can always generate an appropriate operation event on the screen.

  The present invention provides a screen operation system for operating the application program by generating an operation event for an object to be operated in the operation screen for an application program accompanied by display of the operation screen. An operation scenario that defines an object to be operated in the screen, and means for specifying an operation target object that identifies the object to be operated by relative position information in the operation screen defined in the operation scenario A screen operation system is provided.

  In addition, the present invention operates an application program for displaying an operation screen by generating an operation event for an object to be operated in the operation screen in accordance with the description contents of the operation scenario. A program for causing a computer to function as a screen operation system, and when there are a plurality of objects having the same shape as the object to be operated in the operation screen, the program in the operation screen defined in the operation scenario A program for causing a computer to realize a function of acquiring relative position information and a function of specifying an object to be operated based on the acquired relative position information is provided.

  In specifying an object to be operated, it is possible to specify a single object even in a situation where the operation target object is not uniquely determined only by specifying the shape of the single object. Accordingly, it is possible to generate an appropriate operation event on the screen of the client application.

Embodiments of the present invention will be described below with reference to the drawings.
An embodiment of the present invention is directed to a screen operation system for operating an application program by generating an operation event for an object to be operated in the operation screen for an application program accompanied by display of the operation screen. The object to be operated is specified only by specifying the shape of a single object in specifying the object to be operated, having a target object specifying means for specifying the object to be operated by the relative position in the operation screen. The object can be specified even when it cannot be determined uniquely. By having the specific function of the target object, an appropriate operation event can always be generated on the screen of the client application.

  An embodiment of the present invention is directed to a screen operation system that operates an application program by generating an operation event for an object to be operated in the operation screen for an application program accompanied by display of the operation screen. An operation scenario in which an object to be operated in the operation screen is defined, and an operation target object specifying means for specifying the object to be operated by relative position information in the operation screen defined in the operation scenario; It is comprised and comprises. Further, in the screen operation system, the operation object specifying unit is configured such that when the reference image existing in the operation screen is defined in the operation scenario and the distance is defined as the relative position information, the operation screen For each object having the same shape as the object to be operated, the distance information from the reference image is obtained, and the object to be operated is specified based on the distance information. . Further, in the screen operation system, the operation target object specifying means includes an ordinal number for each object having the same shape as the operation target object existing in the operation screen as the relative position information in the operation scenario. When defined, it has a function of specifying the object to be operated based on the ordinal number. The above operation scenario is realized as the operation scenario 112 in FIGS. The means for specifying the operation target object is realized by the target object specifying unit 13 shown in FIGS. 2 and 3 and the image information acquiring unit 15 shown in FIG.

  FIG. 1 is a block diagram showing the configuration of the entire system for performing a load test, and includes a plurality of information terminals (PCs) 1, 1,... On the client side that generate a load and a server 2 to be subjected to the load test. Composed. A plurality of information terminals (PCs) 1, 1,... On the client side are provided with load generation means that use the load generation system 10 and the client application 20 as a unit load generation source for making a processing request to the server 2, respectively.

  In the load test, by adding a large amount of processing requests to the server 2 from the load generating means provided in the plurality of information terminals (PCs) 1, 1,. (CPU etc.) Measure the utilization rate, response time for processing requests, etc.

The configuration of the load generation system 10 shown in FIG. 1 is shown in FIG.
The load generation system 10 performs a screen operation by generating a mouse event or a keyboard event to the client application 20 in the same manner as a human operation. As a result, the client application 20 issues a processing request to the server 2.

  As illustrated in FIG. 2, the load generation system 10 includes a control unit 11, a screen transition determination unit 12, a target object identification unit 13, an operation event generation unit 14, and an image information acquisition unit 15.

  In the load generation system 10 illustrated in FIG. 2, the control unit 11 includes an operation execution control unit 111 and an operation scenario 112 stored in the scenario storage unit.

  The operation execution control unit 111 provided in the control unit 11 controls the screen transition determination unit 12, the target object identification unit 13, and the operation event generation unit 14 according to the description content of the operation scenario 112. The operation scenario 112 defines the contents of a series of pseudo operations to be executed by the server 2 by an operation on the screen. In this embodiment, the shape of a single object is specified in specifying the object to be operated. If the operation target object is not uniquely determined only by specifying the relative positional relationship of multiple objects, it is equipped with a relative position information definition section that makes it possible to specify a single object Is done. A specific configuration of the operation scenario 112 will be described later with reference to FIG.

  The operation execution control unit 111 provided in the control unit 11 issues a screen transition determination request 213 to the screen transition determination unit 12 and receives the screen transition determination response 221 from the screen transition determination unit 12, and the target object An operation target object specifying request 211 is issued to the specifying unit 13, an interface function for receiving the operation target object specifying response 222 from the target object specifying unit 13, and an event generation request 212 is issued to the operation event generating unit 14, And an interface function for receiving an event occurrence response 223 from the operation event generation unit 14.

  The screen transition determination unit 12 issues a screen information acquisition request 215 to the image information acquisition unit 15 in accordance with the screen transition determination request 213 received from the operation execution control unit 111, and sends an image information acquisition response 225 from the image information acquisition unit 15. It has an interface function to accept and send a screen transition determination response 221 to the operation execution control unit 111.

  The target object specifying unit 13 issues a screen information acquisition request 216 to the image information acquisition unit 15 in accordance with the operation target object specification request 211 received from the operation execution control unit 111, and the image information acquisition response 226 from the image information acquisition unit 15. And an interface function for sending the operation target object specifying response 222 to the operation execution control unit 111.

  The operation event generation unit 14 has an interface function for issuing an event 201 to the client application 20 in accordance with the event generation request 212 received from the operation execution control unit 111 and sending an event generation response 223 to the operation execution control unit 111. . The client application 20 sends a processing request 202 according to the event 201 received from the operation event generating unit 14 to the server 2.

  The image information acquisition unit 15 acquires an image information 203 from the client application 20 in accordance with the screen information acquisition request 215 received from the screen transition determination unit 12, and sends an image information acquisition response 225 to the screen transition determination unit 12. An interface function for acquiring the image information 203 from the client application 20 in accordance with the screen information acquisition request 216 received from the target object specifying unit 13 and sending the image information acquisition response 226 to the target object specifying unit 13.

  The operation target object specifying request 211, the screen transition determination request 213, and the image information acquisition responses 225 and 226 are accompanied by image information of a bitmap image (.bmp) included in the operation screen of the client application 20, respectively.

  The configuration of the target object specifying unit 13 shown in FIG. 2 is shown in FIG.

  As illustrated in FIG. 3, the target object specifying unit 13 includes a target object specifying control unit 131, a target object determining unit 132, an image comparing unit 133, and a relative positional relationship analyzing unit 134.

  In accordance with the operation target object specification request 211 received from the operation execution control unit 111, the target object specification control unit 131 sends the image information of the bitmap image associated with the request to the image comparison unit 133, and the screen information acquisition request 216. Is sent to the image information acquisition unit 15 to acquire image information of a bitmap image accompanying the image information acquisition response 226 from the image information acquisition unit 15 and supply the image information to the image comparison unit 133. Realize.

  The image comparison unit 133 compares the image information of each bitmap image acquired by the control of the target object specifying control unit 131, and the image specified by the operation target object specifying request 211 from the operation screen of the client application 20. As a result of the search, a target object candidate table 231 shown in FIG. 16 is created, and the image specified by the operation target object specifying request 211 is stored in the target object candidate table 231 as a reference image (reference object). The object ID and center position (X / Y) coordinates of the reference image are registered, and the object ID and center position (X / Y) coordinates of the searched image (object candidate) are registered. Alternatively, as the search result information, a target object candidate table 232 shown in FIG. 17 is created, and an image (object candidate) according to the relative position information (ordinal number) specified by the operation target object specification request 211 is generated in the target object candidate table 232. ) And the center position (X · Y) coordinates are registered.

  The relative positional relationship analysis unit 134 analyzes the coordinate information of the target object candidate table 231 shown in FIG. 16 created by the image comparison unit 133, calculates the distance to the reference object of each searched image (object candidate), The distance information is associated with the object candidate and registered in the target object candidate table. Alternatively, the target object candidate table 232 shown in FIG. 17 is analyzed, and the position rank information according to the definition content (ordinal number) of the operation scenario 112 is registered in the “position rank”.

  The target object determination unit 132 refers to the target object candidate table, selects an object candidate with the shortest distance from the reference object or one object candidate according to the ordinal rank from the object candidates, and selects the object candidate. The operation target object is specified, and the operation target object is notified to the target object specifying control unit 131.

  Here, the operation of the load generation system 10 will be described with reference to FIGS. 4 to 17 by giving a specific example of the operation screen in the client application 20. Here, the operation will be described by taking a schedule display screen shown in FIG. 4 as an example of the application operation window handled by the client application 20.

  The operation execution control unit 111 provided in the control unit 11 of the load generation system 10 is in accordance with the definition contents of the operation scenario 112, and includes a screen transition determination unit 12, a target object specifying unit 13, an operation event generation provided in the load generation system 10. Each component such as the unit 14 and the image information acquisition unit 15 is controlled.

  The operation execution control unit 111 provided in the control unit 11 sends an operation target object specifying request 211 to the target object specifying unit 13 according to the contents described (defined) in the operation scenario 112, and an operation event generating unit 14, an event generation request 212 is transmitted to the screen transition determination unit 12, and a screen transition determination request 213 is transmitted to the screen transition determination unit 12.

  The screen transition determination unit 12 and the target object identification unit 13 send screen information acquisition requests 215 and 216 to the image information acquisition unit 15 under the control of the operation execution control unit 111, respectively. That is, when the screen transition determination unit 12 receives the screen transition determination request 213 from the operation execution control unit 111, the screen transition determination unit 12 sends a screen information acquisition request 215 to the image information acquisition unit 15. When the target object specifying unit 13 receives the operation target object specifying request 211 from the operation execution control unit 111, the target object specifying unit 13 sends a screen information acquisition request 216 to the image information acquiring unit 15.

  When the image information acquisition unit 15 receives the screen information acquisition request 215 from the screen transition determination unit 12, the image information acquisition unit 15 acquires the image information 203 displayed on the screen by the client application 20 via the operating system (OS), and acquires the acquired image. The included image information acquisition response 225 is sent to the screen transition determination unit 12. Similarly, when the screen information acquisition request 216 is received from the target object specifying unit 13, the client application 20 acquires the image information 203 displayed on the screen, and an image information acquisition response 226 including the acquired image is received as the target object specifying unit 13. To send.

  A specific example of the image information (application operation window) 203 displayed on the screen by the client application 20 at this time is shown in FIG. Here, a schedule display screen 21 in the schedule management system is shown as an example. In the following description of the operation, the image information of the schedule display screen 21 shown in FIG. 4 is acquired as the image information 203 by the image information acquisition unit 15, and the image information of the schedule display screen 21 is the screen transition determination unit 12 and the target. Assume that the data is sent to the object specifying unit 13.

  When the new registration link “9 (Tue)” is clicked on the schedule display screen 21 shown in FIG. 4, the operation screen changes from the schedule display screen 21 shown in FIG. 4 to the schedule registration screen 22 shown in FIG. 5. Further, when the contents are entered in “Time” and “Schedule” on the schedule registration screen 22 and the “Register” button is clicked, the screen transitions to the next screen.

  A procedure for realizing the operation corresponding to the series of screen transitions by the load generation system 10 will be described below.

  The coordinate axes of the schedule display screen 21 shown in FIG. 4 are shown in FIG. Here, an XGA (1024 × 768 dots) application operation window is shown as an example.

  A partial description example of the operation scenario 112 is shown in FIG. As shown in FIG. 7, the operation scenario 112 includes an operation step, screen identification information (screen ID), a bitmap image screen transition determination image, an operation target object image, a reference image, a determination / operation content, Numerical information and relative position information are configured as setting items (definition items).

  The client application 20 is in a state of displaying the schedule display screen 21 shown in FIG. 4 during the screen transition.

  The operation execution control unit 111 reads step 10 of the operation scenario 112. In this step 10, “screen transition determination” is set (defined) as [determination / operation content]. Accordingly, it is determined that the next operation is to determine screen transition. Further, the file name “schedule display.bmp” is set as the “screen transition determination image”. The content (bitmap image) is shown in FIG. The operation execution control unit 111 sends a screen transition determination request 213 including information (bitmap image) I10 of the screen transition determination image shown in FIG.

  Upon receiving the screen transition determination request 213 from the operation execution control unit 111, the screen transition determination unit 12 sends a screen information acquisition request 215 to the image information acquisition unit 15.

  When receiving the screen information acquisition request 215 from the screen transition determination unit 12, the image information acquisition unit 15 acquires the image information (bitmap image) 203 displayed by the client application 20 and includes the image information 203. An acquisition response 225 is sent to the screen transition determination unit 12.

  The screen transition determination unit 12 checks whether there is a portion that matches the image of “schedule display.bmp” in the image information 203 acquired from the client application 20 via the image information acquisition unit 15. The content (bitmap image) of “schedule display.bmp” at this time is the information (bitmap image) I10 of the screen transition determination image shown in FIG.

  In this examination, for example, when there is no matching part because the screen transition is not completed, the screen transition determination unit 12 starts the operation of sending the screen information acquisition request 215 to the image information acquisition unit 15. The same processing as described above is repeatedly executed at an appropriate frequency.

  When the screen transition is completed and a matching part is found, the screen transition determination unit 12 sends a screen transition determination response 221 to the operation execution control unit 111.

  The operation execution control unit 111 reads step 11 which is the next step of the operation scenario 112. In this step 11, “click” is set as [determination / operation content]. Thus, the next thing to do is determined to be a click. Also, the file name “new registration.bmp” is set as the “operation object image”, the file name “9 days.bmp” is set as the “reference image”, and “shortest distance” is set as the “relative position information”. The content (bitmap image) of this reference image “9 days.bmp” is shown in FIG. 9 with the reference I11, and the content (bitmap image) of “new registration.bmp” is shown in FIG. 10 with the reference I12. It shows.

  The operation execution control unit 111 sends an operation target object specifying request 211 to the target object specifying unit 13 according to the contents of the operation scenario 112 described above. The operation target object specifying request 211 is accompanied by the reference image (bitmap image) I11 shown in FIG. 9, the operation target object image (bitmap image) I12 shown in FIG. 10, and the relative position information (shortest distance). is doing.

  In the target object specifying unit 13, when the target object specifying control unit 131 receives the operation target object specifying request 211 from the operation execution control unit 111, it sends a screen information acquisition request 216 to the image information acquiring unit 15.

  When receiving the screen information acquisition request 216 from the target object specifying control unit 131 provided in the target object specifying unit 13, the image information acquiring unit 15 acquires the image information 203 displayed by the client application 20 and acquires the acquired image information 203. The image information 203 is sent to the image comparison unit 133 provided in the target object specifying unit 13. In this case, the schedule display screen 21 shown in FIG. 4 is sent to the image comparison unit 133 as the image information 203.

  The image comparison unit 133 is an image (bitmap image) represented by the file name “new registration.bmp” as the operation target object image from the image information 203 acquired from the client application 20 via the image information acquisition unit 15. Search for the part that matches I12. Here, a plurality of images that match the image (bitmap image) I12 of “new registration.bmp” are respectively searched as target object candidates.

  As a result of the search, the image comparison unit 133 creates a target object candidate table 231 shown in FIG. 16, and each of [object ID], [center X coordinate], and [center Y coordinate] is stored in the target object candidate table 231. Set the information. Thereafter, the image comparison unit 132 takes over the process to the mutual positional relationship analysis unit 134.

  The mutual positional relationship analysis unit 134 analyzes the contents of the target object candidate table 231 shown in FIG. 16 set by the image comparison unit 133, sets distance information in [Distance to Reference Object], and sets the target object determination unit 132. Take over the processing. The processing in the image comparison unit 133 here obtains the distance of each candidate image with respect to the reference image I11 based on the coordinate value between the reference image I11 and each candidate image, and the distance information in the target object candidate table 231 [ Set the distance to the reference object].

  Based on the contents of the target object candidate table 231, the target object determination unit 132 determines that the distance of “candidate 2” set by the image comparison unit 133 as “distance to the reference object” is the shortest distance from the reference image I11. The determination result information is sent to the target object identification control unit 131, and control is passed to the target object identification control unit 131.

  The target object specifying control unit 131 associates the information of the center X coordinate and the center Y coordinate of the candidate 2 according to the information of the determination result received from the target object determining unit 132 with the operation target object specifying response 222, and the operation target object A specific response 222 is sent to the operation execution control unit 111.

  The operation execution control unit 111 receives information about the center X coordinate and center Y coordinate of the candidate 2 and “click” information as the operation / determination content accompanying the operation target object specification response 222 received from the target object specification control unit 131. Is attached to the event generation request 212, and the event generation request 212 is sent to the operation event generation unit 14.

  In response to the event generation request 212 received from the operation execution control unit 111, the operation event generation unit 14 generates an event 201 equivalent to the movement of the mouse to the center X coordinate and the center Y coordinate of the candidate 2 for the client application 20. To do. Further, a mouse click event 201 is generated for the client application 20. Thereafter, an event occurrence response 223 is sent to the operation execution control unit 111.

  In this way, when there are a plurality of operation target objects having the same appearance on the same screen, the “shortest distance” is set in the “relative position information” of the operation scenario 112, and the vicinity position of the operation target is set in the “reference image”. The desired single object can be specified from among a plurality of objects by setting the image in (1). As a result, an appropriate operation event can be generated on the screen of the client application 20.

  Next, the operation execution control unit 111 reads Step 12, which is the next step of the operation scenario 112. In this step 12, “screen transition determination” is set as [determination / operation content].

  Here, the processing in step 12 is executed in the same manner as in step 10 above. The contents (bitmap image) of “schedule registration.bmp” in step 12 are shown in FIG.

  The operation execution control unit 111 reads step 13, which is the next step of the operation scenario 112. In this step 13, “text input” is set as [determination / operation content]. Thus, the next thing to do is determined to be text input. Further, the file name “HH setting.bmp” is set as the “operation target object image”, and “first left and first upper” are set as the “relative position information”. The contents of “HH setting.bmp” are shown in FIG.

  The operation execution control unit 111 sends an operation target object specifying request 211 to the target object specifying unit 13 according to the contents of the operation scenario 112 described above. In the operation target object specifying request 211, the above-mentioned “operation target object image” is the time setting screen (bitmap image) I14 shown in FIG. 12, and the “relative position information” is “first left, first upper” information. Is attached.

  Upon receiving the operation target object specification request 211 from the operation execution control unit 111, the target object specification control unit 131 provided in the target object specification unit 13 sends a screen information acquisition request 216 to the image information acquisition unit 15.

  Upon receiving the screen information acquisition request 216 from the target object specifying control unit 131 provided in the target object specifying unit 13, the image information acquiring unit 15 acquires the image information displayed by the client application 20, and acquires the acquired image. An image information acquisition response 226 including information is sent to the image comparison unit 132. In this case, the image information on the schedule registration screen 22 shown in FIG. 5 is transmitted.

  The image comparison unit 132 selects an image represented by the file name “HH setting.bmp” set as “operation target object image” from the image information acquired from the client application 20 via the image information acquisition unit 15. Search for matching parts. Here, an image that matches the time setting screen I14 shown in FIG. 12 is searched.

  The image comparison unit 133 creates a target object candidate table 232 shown in FIG. 17 as a result of the search, and the target object candidate table 232 includes [object ID], [center X coordinate], and [center Y coordinate]. Set each information. Thereafter, the image comparison unit 132 takes over the process to the mutual positional relationship analysis unit 134.

  The mutual positional relationship analysis unit 134 analyzes the contents of the target object candidate table 232 shown in FIG. 17 set by the image comparison unit 133, sets the position order information from the left in [position order], and sets the target The object determination unit 132 takes over the processing.

  The target object determination unit 132 determines from the contents of the target object candidate table 232 that “Candidate 1” is the first from the left and the first from the top, and information on the determination result is sent to the target object specifying unit 13. Then, control is passed to the target object identification control unit 131.

  The target object specifying control unit 131 associates information on the center X coordinate and center Y coordinate of the candidate 1 according to the determination result information received from the target object determining unit 132 with the operation target object specifying response 222, and specifies the operation target object specifying A response 222 is sent to the operation execution control unit 111.

  When the operation execution control unit 111 receives the operation target object specification response 222 from the target object specification control unit 131, the operation execution control unit 111 receives the information about the center X coordinate and the center Y coordinate of the candidate 1 and the operation scenario. The event generation request 212 is sent to the operation event generation unit 14 with the information “text input” set in 112 [operation / determination content] and the information “9” set in [value]. .

  In response to the event generation request 212 received from the operation execution control unit 111, the operation event generation unit 14 generates an event 201 equivalent to the movement of the mouse to the center X coordinate and the center Y coordinate of the candidate 1 for the client application 20. To do. Furthermore, a text input event 201 of “9” is generated for the client application 20. Thereafter, an event occurrence response 223 is sent to the operation execution control unit 111.

  The operation execution control unit 111 continues the same process as described above for step 14 of the operation scenario 112 and each step after the step. Here, as the [operation object image] of the operation scenario 112, the file name “MM setting.bmp” is set in step 14, the file name “HH setting.bmp” is set in step 15, and the file name “MM setting.bmp” is set in step 16. In step 17, the file name “schedule field.bmp” is set, and in step 18, the file name “registration.bmp” is set. The contents (bitmap image) of the file name “MM setting.bmp” at this time are shown with reference numeral I15 in FIG. 13, and the contents of the file name “schedule column.bmp” are added with reference numeral I16 in FIG. The content of the file name “registration.bmp” is shown with reference numeral I17 in FIG.

  In this way, when there are a plurality of operation target objects having the same appearance on the same screen, “ordinal number” is set (defined) in [relative position information] corresponding to each object image of the operation scenario 112. Thus, it is possible to specify a single object that follows the order of ordinal numbers from among a plurality of objects. As a result, an appropriate operation event can be generated on the screen of the client application 20.

  By the processing operation according to the operation scenario 112 described above, the load generation system 10 can sequentially generate appropriate events to the client application 20 in accordance with the screen transition. A load test for sending a processing request and applying a desired load to the server 2 becomes possible. The above-described specific function of the operation target object can be realized by an application program.

  The application of the present invention is not limited to the load test. For example, in the process of software development, when the software to be developed has a GUI, a so-called function test is performed to verify whether the behavior of the software for performing a predetermined GUI operation is appropriate. . The present invention can also be applied to a use as a tool for operating a GUI instead of a human when performing a function test. In addition, in software having a GUI, an operation abnormality due to a bug may occur. However, the abnormal operation is not always reproduced every time a predetermined GUI operation is performed. Depending on the delicate timing between the internal processing of the software and the GUI operation, the bug may cause an abnormal operation. In order to reproduce the abnormal operation, a predetermined GUI operation may have to be repeated many times. The present invention, which enables the GUI operation to be repeated on behalf of the operator even in such operation for reproducing an abnormal operation, is effective. Furthermore, in order to periodically monitor whether the WWW server is operating normally, the remote WWW browser is operated to access the WWW server, and the monitoring is realized by verifying the result. However, it is necessary to automatically operate the GUI of the WWW browser automatically on behalf of the operator, and the present invention is also effective in this case. Similarly, it is necessary to automatically operate the GUI of the WWW browser on behalf of the operator even when it is desired to periodically repeat the work of accessing information from a remote WWW server and collecting information. Even in this case, the present invention is effective. In addition to the above, the present invention may be effectively used in all cases where there is a need to operate an application having a GUI on behalf of the operator.

The block diagram which shows the structure of the whole system which performs a load test based on embodiment of this invention. The block diagram which shows the structure of the load generation system which concerns on the said embodiment. The block diagram which shows the structure of the target object specific | specification part which concerns on the said embodiment. The figure which shows the specific example of the image information which the client application displays on a screen based on the said embodiment. The figure which shows the specific example of the schedule registration screen which concerns on the said embodiment. The figure which shows the coordinate axis of the schedule display screen which concerns on the said embodiment. The figure which shows the example of a part description of the operation scenario which concerns on the said embodiment. The figure which shows the specific example of the screen transition determination image which concerns on the said embodiment. The figure which shows the specific example of the reference | standard image which concerns on the said embodiment. The figure which shows the specific example of the operation target object image which concerns on the said embodiment. The figure which shows the specific example of the screen transition determination image which concerns on the said embodiment. The figure which shows the specific example of the operation target object image which concerns on the said embodiment. The figure which shows the specific example of the operation target object image which concerns on the said embodiment. The figure which shows the specific example of the operation target object image which concerns on the said embodiment. The figure which shows the specific example of the operation target object image which concerns on the said embodiment. The figure which shows the example of the content of the target object candidate table | surface which concerns on the said embodiment. The figure which shows the example of the content of the target object candidate table | surface which concerns on the said embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Information terminal (PC), 2 ... Server, 3 ... Operator, 10 ... Load generation system, 11 control part, 12 ... Screen transition determination part, 13 ... Target object specific part, 14 ... Operation event generation part, 15 ... Image information acquisition unit, 20 ... client application, 30 ... scenario generation system, 111 ... operation execution control unit, 112 ... operation scenario, 131 ... target object specifying control unit, 132 ... target object determination unit, 133 ... image comparison unit, 134 ... Relative positional relationship analysis unit.

Claims (5)

In a screen operation system for operating the application program by generating an operation event for an object to be operated in the operation screen for an application program accompanied by display of the operation screen,
An operation scenario that defines an object to be operated in the operation screen;
A screen operation system, comprising: an operation object specifying unit that specifies an object to be operated by relative position information in the operation screen defined in the operation scenario.   In the operation scenario, in specifying the object to be operated, if the operation target object is not uniquely determined only by specifying the shape of a single object, the relative positional relationship of a plurality of objects is specified. The screen operation system according to claim 1, further comprising a definition unit for the relative position information that enables the single object to be specified.   The operation target object specifying means is present in the operation screen when a reference image existing in the operation screen is defined for the operation scenario and distance information indicating the relative position information is defined. The distance information from the reference image is obtained for each object having the same shape as the object to be operated, and the object to be operated is specified based on the distance information. 2. The screen operation system according to 2.   When the operation target object specifying means defines an ordinal number indicating the relative position information for each object having the same shape as the operation target object existing in the operation screen for the operation scenario, 4. The screen operation system according to claim 2, wherein an object to be operated is specified based on the ordinal number. As a screen operation system for operating the application program by generating an operation event for the object to be operated in the operation screen according to the description content of the operation scenario for the application program accompanied by the display of the operation screen A program for operating a computer,
A function of acquiring relative position information in the operation screen defined in the operation scenario when there are a plurality of objects having the same shape as the object to be operated in the operation screen;
A program for causing a computer to realize a function of specifying an object to be operated based on the acquired relative position information.

Images