JP6052892B2 - Apparatus for automatically generating GUI stub, method and program thereof - Google Patents

Description

  The present invention relates to a technique for supporting creation of a control program for operating a GUI of software.

  There are many interfaces for controlling software. As an interface provided by most software, there is a graphical user interface (GUI) that is assumed to be operated by a human. In recent years, an application has been developed in which the GUI is controlled by other software (agent), not a person, so that the software including the GUI can be operated without a person to automate the work on the software.

  In order to automatically operate the GUI, it is necessary to create a GUI control program. As conventional techniques for creating / executing a control program, tools such as UMS, UWSC, and AutoIt are known (see Non-Patent Documents 1, 2, and 3). According to these tools, it is possible to create the control program by describing the contents of the automatic operation to the GUI in various formats such as a macro / script and a flowchart.

  However, the conventional tool described above has a problem that an actual system, that is, software to be automatically operated (hereinafter referred to as target software) is required to create a control program.

  At the stage of creating the control program, it is necessary to repeat trial and error and correction of the control program (debugging work) in order to obtain the desired automatic operation, but such debugging work is performed on the target software. Risk is great. The reason is that a control program that has not been completed is executed many times in the process of debugging, and an undesired operation such as deleting data by mistake may be executed / reflected in the target software. Because there is. If the target software is a running business system, such a risk is unacceptable.

  Therefore, the present invention solves the problem of enabling debugging by creating a control program for operating the GUI of the target software without affecting the target software.

  In order to achieve the above object, the present invention proposes an automatic GUI stub generation apparatus, method and program thereof.

Here, the “GUI stub” refers to a program that can be used in place of the target software in creating or debugging a control program,
1. It has the same screen (appearance and structure) as the target software.
2. Perform the same screen transition as the target software.
3. The operation event observed when operating the GUI stub is the same as when operating the target software.
It has the requirement that.

In particular,
An apparatus for automatically generating a GUI stub, which is a program that can be used in place of the target software, when creating or debugging a control program that automates the work on the target software by operating the GUI of the target software ,
In response to an operation by the operator on the target software, an operation log acquisition unit that acquires an operation log including the operation content and a GUI state including a GUI structure information and a pair of image information of the target software at that time;
An operation log recording means for recording the acquired operation log as a sequence in an arbitrary format in which the GUI state is represented as a node and the operation content as an edge;
Transition destination selection means for selecting and outputting the GUI state to be reconfigured next from the series recorded in the operation log recording means or the operation contents observed for the GUI stub together with the series;
Using the output GUI state as input, search for all GUI parts included in the structure information, read individual information of each GUI part, create GUI parts to match the individual information, and build GUI stub GUI construction means to
Operation observing means for observing operation contents for the purpose of creating or debugging a control program for the constructed GUI stub;
It is characterized by comprising information output means for outputting at least the operation contents for the GUI stub and the structure information of the GUI stub, which are necessary for creating or debugging the control program.

  According to the present invention, it is possible to eliminate the influence on the target software by automatically generating a GUI stub from the operation log of the target software and creating or debugging the control program on the GUI stub.

  As a preferred embodiment of the present invention, the GUI construction unit performs processing for complementing GUI construction by image information, processing for converting GUI implementation, and processing for changing the GUI appearance, using parameters as input. The thing characterized by including is mentioned.

  In other words, not only the same GUI as the target software is constructed, but also a GUI implemented in Java (registered trademark) is converted into a web GUI according to the parameters, or it is not necessary for creating a control program. It is possible to improve the reproducibility by combining a GUI construction with changes such as deletion of GUI parts, or in combination with image information when the construction is incomplete.

  Similarly, as a preferred embodiment of the present invention, when the operation observation unit observes an operation on the GUI converted or changed by the GUI construction unit, the operation content corresponds to the GUI before conversion / change. And the like, which include a process of converting the operation contents to output and outputting them. As a result, the same operation content can be output to the transition destination selection unit regardless of the presence / absence of GUI conversion / change, and a control program can be created for a GUI stub constructed by adding conversion / change. Even when debugging is performed, the information output unit outputs the operation contents and GUI state to the automatic operation target software before conversion / change, and the automatic operation target software removes the influence of conversion / change. An operating control program can be created.

  Similarly, a preferred embodiment of the present invention includes a process of estimating a transition destination when there is an unknown transition destination that is not recorded in the operation log in the transition destination selection unit. Things. That is, it is possible to create a GUI stub from an insufficient operation log by selecting a GUI state that is inferred to be the same as the transition destination from the similarity between the operation content and the GUI state.

  According to the present invention, it is possible to apply a GUI stub without using the target software in the creation or debugging of a control program for automatically operating the GUI of the target software, thereby avoiding the influence on the target software. It becomes. Also, by inputting parameters according to the situation at the time of creation / debugging of the control program, it is possible to create a GUI stub to which conversion / change has been added and to estimate the GUI state of an unknown transition destination. As a result, it is possible to create a control program on a platform different from the target software and to create a GUI stub from an insufficient operation log, and to support the creation / debugging of the control program more strongly.

The block diagram which shows an example of embodiment of the GUI stub automatic generation apparatus of this invention Explanatory drawing which shows an example of the structure information of GUI of object software Explanatory drawing showing an example of a series representing operation logs Explanatory drawing of operation in transition destination selection part Operation flow chart of the GUI stub automatic generation apparatus of the present invention Operation flow chart of control program creation / debug tool using GUI stub automatic generation device of the present invention Explanatory drawing which shows an example of a GUI conversion process Explanatory drawing which shows an example of a GUI change process Explanatory drawing which shows an example of a GUI image complement process Explanatory drawing which shows an example of a transition destination estimation process Explanatory drawing which shows an example of a context estimation process Explanatory drawing which shows an example of operation conversion processing

  An example of an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 shows an example of an embodiment of a GUI stub automatic generation apparatus according to the present invention together with target software and an operating environment of a control program creation / debug tool.

  That is, the target software A to be automatically operated operates on the computer 10, and the control program creation / debug tool B which is a tool for creating or debugging a control program for automatically operating the target software A is displayed on the computer 20. In this case, the GUI stub automatic generation device 30 according to the present embodiment provides the GUI stub C instead of the target software A to the control program creation / debug tool B to create the control program or Provide services that support debugging.

  Here, various forms of the control program creation / debugging tool B are conceivable, for example, a tool for creating a control program using a script language or a flowchart using the structure information and image information of the target software A main body.

  The computer 10 includes an OS 11 as an execution environment of the target software A, an input device 12 such as a keyboard and a mouse as a user interface to an operator 41 who operates the target software A, and a display 13 as a display device. The computer 20 has an OS 21 as an execution environment for the control program creation / debug tool B and an input device such as a keyboard and a mouse as a user interface to the control program creator 42 who operates the control program creation / debug tool B. 22 and a display 23 which is a display device.

  Note that OS11 and OS21 may be the same or different. Although the target software A and the control program creation / debug tool B are operating on different computers here, they may be operating on the same computer.

  Hereinafter, the GUI stub automatic generation device 30 according to the present embodiment will be described in detail.

  That is, the GUI stub automatic generation device 30 includes an operation log acquisition unit 31, an operation log recording unit 32, a transition destination selection unit 33, a GUI construction unit 34, an operation observation unit 35, and an information output unit 36, as shown in FIG. A GUI stub C is created.

  The GUI stub automatic generation device 30 can also be realized by a computer and a program, can be operated by a computer remotely connected to the computers 10 and 20 via a network, and can also be operated by the computer 10 or 20. Is possible. Furthermore, the functional units 31 to 36 do not have to exist on one computer, and can operate in a distributed environment.

  The operation log acquisition unit 31 acquires an operation log when the target software A is actually operated. Specifically, using the functions provided by the OS, the Web browser, and the system library associated therewith, when the operator 41 operates the target software A, the operation contents and the GUI of the target software A at that time are displayed. An operation log including a GUI state composed of a pair of structure information and image information is acquired. Examples of the system library include Windows (registered trademark) Accessibility API, UI Automation, and DOM in the case of a Web browser system, and any other can be used.

  FIG. 2 shows an example of structure information, which is a hierarchical structure with GUI parts such as buttons and edits as nodes, and holds various individual information such as position, size, name, etc. for each GUI part (each node). Yes. The image information is image data obtained by capturing the entire GUI of the target software A displayed on the display 13. The operation contents of the operator 41 are a click on the GUI of the target software A displayed on the display 13, an input from the keyboard, an operation on each GUI component, etc. (button pressed, check box on / off, etc.) In addition, various user operations from the input device 12 are acquired as operation contents.

  The operation log recording unit 32 records the operation log acquired by the operation log acquisition unit 31. Specifically, the acquired operation log is recorded as a sequence in an arbitrary format in which the GUI state is represented as a node and the operation content is represented as an edge.

  FIG. 3 shows an example of a series representing an operation log. Here, an example of a series in each form of a state transition diagram, a list group, and a multitree is shown. The series represents the GUI state (S0, S1, S2...) As a node and the operation content (O1, O2, O3...) As an edge.

  The transition destination selection unit 33 selects and outputs the GUI state to be reconfigured next from the operation log recorded in the operation log recording unit 32 or the observed operation on the GUI stub C together with the operation log. Specifically, a series as an operation log recorded in the operation log recording unit 32 or an operation content from the operation observation unit 35 described later is input as an input, and a GUI state to be reconfigured next is selected and output.

  The transition destination selection unit 33 will be described with reference to FIG. As shown in FIG. 4, the edge (operation content) extending from the currently reconfigured GUI state is searched for and matches the operation content observed and output by the operation observation unit 35. If one is found, the GUI state ahead of the edge is selected and output as the GUI state to be reconfigured next.

  Further, the transition destination selection unit 33 includes a transition destination estimation process for estimating the transition destination GUI state when there is no transition destination corresponding to the operation content from the operation observation unit 35. The transition destination estimation process will be described later.

  The GUI construction unit 34 constructs a GUI stub C having the same appearance as that of the target software A with the GUI state output from the transition destination selection unit 33 as an input. Specifically, the GUI state output from the transition destination selection unit 33 is input, and all nodes (GUI parts) in the structure information are searched, and individual information such as the position, size, and name of each GUI part is read, A GUI component is created so as to match the individual information, and a GUI stub C is constructed.

  Further, the GUI construction unit 34 receives a parameter for creating or debugging a control program as input, and performs a GUI conversion process for converting the type of GUI entity, and a GUI for changing the constructed GUI entity from the appearance of the target software A. A change process and a GUI image complement process for complementing the GUI with image information when the structure information is incomplete are performed. The GUI image complement processing, GUI conversion processing, and GUI change processing will be described later.

  The operation observation unit 35 observes an operation for creating or debugging a control program for the constructed GUI stub C, and outputs the operation content. In detail, it is the same as the acquisition of the operation content in the operation log acquisition unit 31, and clicks from the mouse, input from the keyboard, operations on individual GUI components, etc. (buttons are pressed, check boxes are turned on / off, etc.) In addition, various operation contents from the input device 22 are acquired. Further, when the constructed GUI stub C is accompanied by a GUI conversion process or a GUI change process, an operation conversion process is performed. The operation conversion process will be described later.

  The information output unit 36 outputs information necessary for creating or debugging a control program. Specifically, in response to a request from the control program creation / debugging tool B, the operation contents applied to the GUI stub C and information on the GUI state at that time are output as information necessary for the creation or debugging of the control program. .

  Next, the operation of the GUI stub automatic generation device 30 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 5, first, the operation log acquisition unit 31 receives the structure information and image information of the GUI of the target software A, and the operation contents when the operator 41 operates the target software A for which a GUI stub is desired to be created. Obtained as an operation log (step s1).

  Next, the operation log recording unit 32 sequentially records the operation log, and creates a sequence in an arbitrary format in which the GUI structure information and image information pair is set to the GUI state, the GUI state is a node, and the operation content is an edge (step) s2). At this time, the GUI state when the target software A is activated is set as an initial state.

  Next, the transition destination selecting unit 33 selects and outputs the GUI state to be reconfigured next by using the output of step s5 (operation contents to the GUI stub C) and the sequence of the operation log recording unit 32 as input (initially) The initial state is output) (step s3).

  Next, after accumulating the operation log for an arbitrary time, the GUI construction unit 34 reads the GUI state (initial state) output from the transition destination selection unit 33, and constructs the GUI stub C (step s4).

  Next, the operation observation unit 35 observes an operation for the purpose of creating or debugging a control program on the constructed GUI stub C and outputs the contents (step s5).

  Next, in response to a request from the control program creation / debug tool B, the operation content or GUI state is output to the control program creation / debug tool B (step s6). Thereafter, steps s3 to s6 are repeated until completion.

  Next, the operation of the control program creation / debug tool B using the GUI stub automatic generation apparatus of the present invention will be described with reference to FIG. Here, a case where the control program creator 42 executes an automatic operation control program for the target software A on the computer 20 as the control program creation / debug tool B will be described as an example.

  As shown in FIG. 6, the control program creation / debugging tool B applies an operation to the GUI stub C created by the GUI stub automatic generation device 30 for the purpose of creating or debugging the control program (step s11).

  Next, the information output unit 36 of the GUI stub automatic generation device 30 controls information necessary for creating or debugging a control program such as the operation contents applied to the GUI stub C and the GUI state at that time. The data is output to the program creation / debug tool B (step s12).

  The control program creation / debug tool B creates or debugs the control program based on the information in step s12 (step s13). The details of the creation of the control program or the debugging process are the same as those well known in the art and will not be described in detail.

  Finally, the GUI stub C transitions to the next screen according to the operation content, and thereafter steps s11 to s13 are repeated.

  By the way, in the present invention, the GUI construction unit 34 can add any one or more of the GUI conversion process, the GUI change process, and the GUI image complementing process with the parameters at the time of creating or debugging the control program as an input. .

  Here, the GUI conversion processing means that when implementation conversion is given as a parameter to the GUI stub to be constructed, the structure information is converted according to the parameter, and the GUI stub is constructed with a different implementation from the target software. The process to do. The GUI change process is a parameter that is added to a GUI stub to be constructed when parameters are given for GUI changes such as deletion of GUI parts that are not required when creating / debugging a control program and changes in size and arrangement thereof. This is a process of adding changes according to. Furthermore, the GUI image complementing process is a method of clipping and arranging image information corresponding to a part of an incomplete part of the structure information when the supplementation by image information is given as a parameter. This is a process that complements the appearance of and improves the recall.

  These include creating a control program or debugging on a terminal or platform different from the actual target software, removing or changing GUI parts unnecessary for creating or debugging a control program, and GUI stub recall. It is possible to increase.

  Specifically, as an example of the GUI conversion process, FIG. 7 shows a method of applying a structure information conversion map for each conversion target implementation. As shown in FIG. 7, the GUI component information included in the structure information can be converted into an HTML expression or an XAML expression using a conversion map, so that the GUI can be constructed with different implementations.

  As an example of the GUI change process, FIG. 8 shows a method for defining a change using a style sheet. As shown in FIG. 8, for example, by describing the definition of the GUI part to be deleted and the definition of the GUI part to be changed in size and arrangement in the style sheet, the change is reflected in the GUI construction. Can do.

  As an example of the GUI image complementing process, FIG. 9 shows a method based on comparison between the constructed GUI capture image and the image information of the operation log. Depending on the GUI component, the structure information may not be completely acquired. For example, structural information cannot be acquired for GUI parts that are uniquely created, or for some GUI parts that are mounted using Java (registered trademark). For this reason, as shown in FIG. 9, a GUI stub created only from the structure information may be deficient, resulting in a difference from the image information. In the GUI image complementing process, it is possible to increase the reproducibility as an appearance by recognizing a missing part of a GUI constructed by comparing two images and embedding the clipped image information in the missing part.

  Furthermore, transition destinations of all GUI states of the target software are rarely recorded as operation logs, and when creating a GUI stub, a transition destination that does not exist in the GUI state transition diagram may be required. In such a case, the transition destination selection unit 33 compensates for deficiencies in the operation log by estimating and outputting a transition destination that is not recorded.

  As an example of the process of estimating an unknown transition destination from the recorded operation log, it is estimated that the same from the similarity of the GUI state in the operation log and the observed operation content as shown in FIG. There is a method for outputting a GUI state.

  As a method for obtaining the similarity of the GUI state, any existing method such as similarity based on the edit distance of the tree structure can be applied to the structure information, and the image information is similar using an average of pixel values or a histogram. Any existing method such as degree can be applied. Further, the similarity of operation contents can be arbitrarily defined according to the type of each operation. For example, the similarity between the operation content “click coordinates (X, Y)” and the operation content “click coordinates (X ′, Y ′)” is (X, Y) and (X ′, Y ′). It can be defined by the similarity based on the Euclidean distance. In addition, it is also possible to define that similarity is allowed only when it completely matches, and similarity is not allowed otherwise.

  In addition, in estimating the transition destination, in addition to the method based on the similarity, a person can teach (add) an edge that does not exist in the recorded operation log in advance.

  In addition, in the case of complex transitions in which the transition destination is determined according to the past operation content or GUI state sequence, the transition destination is determined by context estimation processing for estimating the transition destination by giving context information in advance. decide. Specifically, as shown in FIG. 11, when a specific path information (condition) and a specific transition destination (operation) are given as context information, a specific transition destination when such a specific path is observed. Here, it is possible to display a warning on the GUI stub as a prohibited operation.

  Furthermore, in the present invention, when the GUI conversion process or the change process is included, a GUI that is different (changed) from the information recorded in the target software and the operation log can be constructed. As a result, the operation performed on the GUI to which the conversion or change process has been added changes from the operation on the target software or the operation recorded in the operation log.

  For example, as shown in FIG. 12, when the button is changed by GUI change processing and constructed, when the operation is performed on the button, the operation “click coordinates (X ′, Y ′)” is performed. Although observed by the observation unit 35, what is recorded on the actual software or in the GUI state transition diagram of the operation log is an operation of “clicking coordinates (X, Y)”.

  In the operation conversion process, a change in the operation content associated with the GUI conversion process or the GUI change process is removed, and the operation is converted into an operation stored in the target software or operation log. Specifically, the operation observation unit 35 refers to the information of the conversion map and style sheet used in the GUI conversion process or the change process, the GUI structure information before or after the conversion, and the converted or changed information after the conversion. By acquiring the correspondence with the GUI structure information, the operation content is converted to the one before conversion or before change.

  As described above, according to the GUI stub automatic generation device according to the present invention, the control program can be created or debugged in the GUI stub automatically generated from the operation log. It is possible to create or debug a control program that eliminates such risks. In addition, it is possible to create GUI stubs on different platforms and GUI stubs with different appearances by conversion processing, change processing, image complement processing, and operation conversion processing in GUI construction. In addition, it is possible to create GUI stubs from insufficient operation logs by transition destination estimation processing. Thereby, not only the risk reduction in the creation or debugging of the control program but also the control program development can be strongly supported.

  10, 20: Computer, 11, 21: OS, 12, 22: Input device, 13, 23: Display, 30: GUI stub automatic generation device, 31: Operation log acquisition unit, 32: Operation log recording unit, 33: Transition Destination selection unit, 34: GUI construction unit, 35: Operation observation unit, 36: Information output unit, 41: Operator, 42: Control program creator, A: Target software, B: Control program creation / debug tool, C: GUI stub.

Yokose, Tanaka, Toyota, Horita, Inoue, Sugimoto, “Overview and Application Examples of Terminal Operation Automation Software UMS”, IEICE Technical Report, vol.111, no.488, ICM2011-73, March 2012, pp.149 -154 "Windows automation software UWSC", [online], umiumi, [searched August 5, 2013], Internet <URL: http://www.uwsc.info/> "Automated script language AutoIt", [online], AutoIt Consulting Ltd., [August 5, 2013 search], Internet <URL: http://www.autoitscript.com/site/>

Claims (10)

An apparatus for automatically generating a GUI stub, which is a program that can be used in place of the target software, when creating or debugging a control program that automates the work on the target software by operating the GUI of the target software ,
In response to an operation by the operator on the target software, an operation log acquisition unit that acquires an operation log including the operation content and a GUI state including a GUI structure information and a pair of image information of the target software at that time;
An operation log recording means for recording the acquired operation log as a sequence in an arbitrary format in which the GUI state is represented as a node and the operation content as an edge;
Transition destination selection means for selecting and outputting the GUI state to be reconfigured next from the series recorded in the operation log recording means or the operation contents observed for the GUI stub together with the series;
Using the output GUI state as input, search for all GUI parts included in the structure information, read individual information of each GUI part, create GUI parts to match the individual information, and build GUI stub GUI construction means to
Operation observing means for observing operation contents for the purpose of creating or debugging a control program for the constructed GUI stub;
An apparatus for automatically generating a GUI stub, comprising information output means for outputting at least the operation contents for the GUI stub and the structure information of the GUI stub, which are necessary for creating or debugging a control program. Control as input parameters for creating or debugging of your program, if the implementation of the conversion on the GUI stubs constructed is given as a parameter, the structure information is converted in accordance with the parameter, the target software and different implementations The GUI stub automatic generation apparatus according to claim 1, further comprising GUI construction means for performing GUI conversion processing for constructing a GUI stub. Control as input parameters for creating or debugging of your program, delete a GUI component to be unnecessary when creating or debugging of the control program, or if the parameter is given for GUI change is a change in the size or arrangement, constructed 2. The GUI stub automatic generation device according to claim 1, further comprising a GUI construction unit that performs a GUI change process for changing a GUI stub according to the parameter. 3. As input parameters for creating or debugging the control program, if there is complementary by the image information it is given as a parameter, and clipping the image information corresponding to the portion relative to incomplete portions of the structure information arrangement The GUI stub automatic generation device according to claim 1, further comprising GUI construction means for performing GUI image complement processing that complements the appearance of the GUI and increases the reproduction rate. Relative GUI stub constructed include pre Symbol GUI conversion processing, if you observe the creation or debugging operation for the purpose of the control program, and the structure information of the previous GUI that will be converted, it is converted An operation observation unit for performing an operation conversion process for converting the operation content into the operation content to the GUI before conversion by comparing with the structure information of the GUI after the operation is further provided. 2. An apparatus for automatically generating a GUI stub according to 2 . Relative GUI stub constructed include the GUI change processing, if the creation or debugging of the control program was observed operations for the purpose, and the structure information of the previous GUI that changes are applied, changes are added It is to comparing the structure information of a GUI after et been, and further comprising a manipulation monitoring unit for performing operation converting process for converting the operation contents to the operation contents to change the previous GUI The GUI stub automatic generation device according to claim 3 . When the observed operation content for the GUI stub is input and the transition destination corresponding to the operation content is not recorded in the operation log, the similarity of the GUI state in the operation log or the similarity of the observed operation content , based on some have teaching given before thing to claim 1, further comprising a transition destination selection unit which performs transition destination estimation processing and outputting the estimated GUI state of the transition destination The GUI stub automatic generation device according to any one of claims 6 to 7. When the context information including the specific path information and the specific transition destination is input together with the observed operation content for the GUI stub, the specific transition destination when the operation content or the GUI state series according to the path information is observed The GUI stub automatic generation device according to any one of claims 1 to 6 , further comprising a transition destination selection unit that performs a context estimation process for estimating a transition destination as a transition destination. A method for automatically generating a GUI stub, which is a program that can be used in place of the target software, when creating or debugging a control program that automates the work on the target software by operating the GUI of the target software. ,
A first step of acquiring an operation log including an operation content and a GUI state formed by a pair of GUI structure information and image information of the target software at the time of an operation by the operator on the target software;
A second step of recording the acquired operation log as a sequence in an arbitrary format in which the GUI state is represented as a node and the operation content as an edge;
A third step of selecting and outputting the GUI state to be reconstructed next from the recorded sequence or the operation content observed with the GUI stub together with the recorded sequence;
Using the output GUI state as input, search for all GUI parts included in the structure information, read individual information of each GUI part, create GUI parts to match the individual information, and build GUI stub A fourth step to:
A fifth step of observing operation contents for the purpose of creating or debugging a control program for the constructed GUI stub;
A method for automatically generating a GUI stub, comprising: a sixth step for outputting at least the operation contents for the GUI stub and the structure information of the GUI stub, which are necessary for creating or debugging a control program. The program for functioning a computer as each means of the automatic generation apparatus of the GUI stub in any one of Claims 1 thru | or 8 .

Images