JP2014102634A - Method and apparatus for supporting user operation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To support correct operation through GUI, for an application which may cause trial-and-error operation and includes non-stereotyped operation with fluid screen transition.SOLUTION: When an operation support request is inputted from a user, an operation information acquisition unit 7 acquires an operation screen at the point of time (present) and GUI constitution information and an assumed screen generation unit 8 selectably displays images of respective states of a GUI component capable of obtaining two or more visually different states out of GUI components on the present operation screen on the basis of a GUI component state management table and generates an assumed screen by changing the visual states of the GUI components on the present operation screen to a state selected by the user. A retrieval key is generated from a GUI component XML tree corresponding to the assumed screen and an operation example storage unit 1 is retrieved by using the operation key.

Description

  The present invention relates to a technique for supporting a user (operator) operation on an application program operating on a computer, which realizes a business system in a company.

  Most of recent application programs (hereinafter referred to as applications) are provided with a graphical user interface (hereinafter referred to as GUI). In the GUI, an instruction or input to an application is performed by operating a GUI component (GUI component) such as a button or a text box on the screen displayed on the display device. The operation here specifically means mouse click or keyboard input by the user.

  At this time, when an operation is performed on a GUI component on the screen, the screen (GUI screen) changes. This is called screen transition. In the screen transition, the button may be displayed in reverse video, the text in the window may change, or another window may be opened. One work in an application is normally completed when a correct operation is performed on a plurality of predetermined screens displayed one after another by screen transition.

  For routine work in which the order of operations is determined in advance and the screen transition is fixed, various support methods for performing correct operations on individual screens have been developed and proposed. However, there is no support method for performing correct operations on individual screens for atypical work that involves trial and error and fluid screen transitions, and operations that can be used for heuristic judgment while predicting the expected screen A system that can search for cases is desired.

  In elementary routine work, operation support has been provided to users by an operation manual, work manual, help function, annotation function, check function, and the like. In these methods, the operation support information presented to the user is fixed with respect to the input information. In order to cope with more advanced work, it is necessary to create data for a large amount of operation support according to various cases. Even if a large amount of operation support data is created, when searching for information required by the user from a large amount of operation support data, in order to specify the exact operation support data desired by the user, You must enter search key information.

  In recent years, the basic routine work targeted by the operation manual and the like as described above has been automated and the amount thereof is decreasing.

  On the other hand, when a high-level operation such as sudden failure recovery, failure analysis, or network design is required, it is difficult to cope with a fixed operation manual as described above. In such work, the user needs to respond flexibly to a given situation. Up to now, users have combined them based on various past experiences according to the situation, and the user has made judgments according to the situation under the guidance of other high-skilled operators.

[Conventional technology]
1. Search text using text as search key (Namazu)
If a character string included in an operation example is a search target, a search using a character string as a search key can be easily performed by using a text search system (see Non-Patent Document 1).

2. Search to search for images using text as search key (Filckr)
When a character string is used as a search key, if the search target is an image, it is necessary to associate some character string with the image data. For example, in Flickr, a photo sharing service on the Web, the shooting date / time, location, equipment, and shooting settings extracted from the tag (character string) that the user inputs at the time of posting or Exif data that is additional information of the image Etc. are associated with each image. A list of corresponding images (for example, images having tags “Everest” and “mountain”) by inputting a character string search key (for example, “Everest mountain”) for this character string data. Can be obtained (see Non-Patent Document 2).

3. Search for an image using the image as a search key (CBIR: Content-Based Image Retrieval)
A method of extracting a feature vector of an image such as a histogram or data size and performing a search by comparing with a feature vector of an image to be searched. Problem: Processing load is heavy, and processing takes time when searching from a large amount of data. It is difficult to search using characters or symbols or semantically (see Non-Patent Document 3).

4). Searching for an image using a combination of a character string and an image as a search key First, it is possible to perform a search by image and further search the result narrowed down by a character string. With this method, if you search for application manuals or operation examples, you can narrow down the information you want more precisely based on the captured image that shows the situation you want to search and the keywords that the user is explicitly aware of. it can. Problem: Since entry of a character string depends on the user, the user must input an appropriate character string.

5. Search by image (Google)
The current Google image search provides a search service using text as a search key and search using an image as a search key. However, the mechanism for realizing a search using an image as a search key is to first generate a character string representing the image by matching the image given as the search key with an original image collection, etc., and then use that character string as the search key. The search execution result is presented. Problem: A character string representing an image is user-dependent, and an appropriate character string needs to be associated with the image.

6). Search using user terminal screen and screen information A method of searching for an operation example using a user operation screen and screen information as a search key has been proposed (Japanese Patent Application No. 2012-149522). Here, the problem of the search key is overcome by using the user's operation screen and screen information as the search key. Problem: An operation example similar to the user's operation screen is searched instead of an operation example similar to the screen assumed by the user.

  The conventional techniques described above have the following problems.

  1. While checking the current operation screen and the assumed screen by the user, and the assumed screen and the operation example, it was not possible to heuristically search for an operation example that serves as a reference for the operation.

  2. Although it is possible to search using the currently displayed screen (As is) as a search key, it has not been possible to perform a search on a screen (To Be) assumed by a user who does not know the technical terms of the GUI component.

  3. When a database is searched using an image as a search key, a semantic search specialized for GUI parts such as buttons, text boxes, and radio buttons cannot be performed.

  An object of the present invention is to enable support of correct operation through a GUI for an application including an atypical work in which the operation involves trial and error and the screen transition is fluid.

  In order to achieve the above object, the present invention provides a method for assisting a user operation through a GUI for an operation target application operating on an arbitrary computer, the network being connected to the arbitrary computer or the arbitrary computer via a network. Other connected computers acquire operation case log data in the operation target application, and include operation case information including a plurality of time-series operation screens indicated in the operation case and a GUI part XML tree of the operation screens. GUI components that can take two or more visually different states based on the operation case information collection step for storing the operation case information in the operation case storage unit, the operation screen in the log data, and the GUI component XML tree of the operation screen. G for managing component images and attribute values that control the state A GUI component state management table generating step for generating an I component state management table, an operation information acquiring step for acquiring image information and GUI component configuration information of the current operation screen when an operation support request is input from the user, Based on the GUI component state management table, an assumed screen generating step for generating an assumed screen from the current operation screen and the state of the GUI component on the current operation screen, and an operation screen similar to the assumed screen from the operation case storage unit A search process for searching operation case information including, a search result display process for displaying any one of time-series operation screens in the searched operation case information in a selectable manner by the user, and operation case information selected by the user The collation process that collates the current operation screen with the current operation screen in units of GUI parts and displays a screen that clearly shows the inconsistencies. · Characterized in that it results executes the display process.

  In order to achieve the above object, the present invention is a device that supports user operation through a GUI for an operation target application that operates on an arbitrary computer, and stores various types of information related to operation cases in the operation target application. GUI component state management that stores a GUI component state management table that manages an image of the GUI component and an attribute value that controls the state of a GUI component that can take two or more visually different states. Acquire log data of operation cases in the table storage unit and the operation target application, and operate the operation case information including a plurality of time-series operation screens shown in the operation cases and a GUI part XML tree of the operation screens. The operation case information collection unit to be stored in the case storage unit and the log data Based on the image information of the operation screen and the GUI part XML tree, a GUI part state management table for managing the GUI part image and attribute values for controlling the state is generated for GUI parts that can take two or more visually different states. And a GUI component state management table generation unit that stores the GUI component state management table storage unit, and operation information acquisition that acquires image information and GUI component configuration information of the current operation screen when an operation support request is input from the user An assumption screen generation unit that generates an assumption screen from the current operation screen and the state of the GUI component on the current operation screen based on the GUI component state management table in the GUI component state management table storage unit, and the operation A search unit for searching operation case information including an operation screen similar to the assumed screen from the case storage unit; The search result display unit that displays any of the time-series operation screens in the searched operation case information that can be selected by the user, and the operation screen selected by the user and the current operation screen are collated in units of GUI parts. And a collation processing / result display unit for displaying a screen clearly showing the inconsistent portions.

According to the present invention,
1. There is no need to include text information for search in the image.
2. The user does not need specialized knowledge about GUI parts and can present a screen assumed for the user without inputting text as a search key.
3. The user can obtain a search result more suited to the user's image by performing the GUI part image assuming the search key.
4). By setting search conditions, the target of matching can be changed flexibly for the user.
Such effects can be achieved.

The block diagram which shows an example of embodiment of the user operation assistance apparatus of this invention Explanatory drawing which shows an example of a GUI component state management table Explanatory drawing which shows the rule of GUI component state management table generation Operation flowchart in user operation support device of the present invention Explanatory diagram showing the flow from operation support request to search key generation Explanatory drawing which shows an example of the screen at the time of operation support request Explanatory drawing which shows an example of assumption screen generation Explanatory drawing which shows an example of a search result display screen Explanatory drawing showing an example of a time series display screen Explanatory diagram showing the flow of verification processing Explanatory drawing which shows an example of a collation result display screen

  FIG. 1 shows an example of an embodiment of a user operation support apparatus according to the present invention, here an example realized on a well-known computer. In the figure, 1 is an operation example storage unit, and 2 is a GUI component state management table storage. , 3 is a log data acquisition unit, 4 is a data import unit, 5 is a related table generation unit, 6 is a GUI component state management table generation unit, 7 is an operation information acquisition unit, 8 is an assumed screen generation unit, and 9 is a GUI component. An XML tree generation unit, 10 is a search execution unit, 11 is a search result display unit, and 12 is a collation processing / result display unit.

  The operation case accumulation unit 1 is for accumulating various types of information related to operation cases in the operation target application, and includes an image image accumulation unit 1a that accumulates image information of a plurality of time-series operation screens in each operation case, A GUI component XML tree storage unit 1b for storing a GUI component XML tree on the operation screen, and a related table storage unit 1c for storing a related table indicating the relationship between the image information on the operation screen and the GUI component XML tree are provided. .

  The GUI component state management table storage unit 2 is a GUI component state management table that manages an image of a GUI component and an attribute value that controls the state of a GUI component that can take two or more visually (image-like) states. Accumulate.

  The log data acquisition unit 3 provides a library for generating and acquiring GUI information on screens for automatic tests related to various operation cases in the application to be operated, for example, a person who has a visual handicap or an automatic test. (Registered trademark) This is executed using UI Automation technology, and at that time, image information and GUI part XML trees of a plurality of time-series operation screens generated are acquired as log data.

  The data import unit 4 stores the image information of the operation screen and the GUI part XML tree in the log data in the image image storage unit 1a and the GUI part XML tree storage unit 1b of the operation case storage unit 1, respectively. The related table generating unit 5 generates a related table indicating the relationship between the image information of the operation screen in the log data and the GUI part XML tree, and stores it in the related table storage unit 1c.

  The log data acquisition unit 3, the data import unit 4, and the related table generation unit 5 constitute an operation case information collection unit referred to in the claims.

  The GUI component state management table generation unit 6 uses a GUI component of the GUI component that can take two or more visually different states based on the image information of the operation screen in the log data and the GUI component XML tree. A GUI component state management table that manages images and attribute values that control the state is generated.

  FIG. 2 shows an example of a GUI component status management table. Each GUI component type has two or more visually different images, properties (attributes) that control the status, and the images. The value of the corresponding property is registered.

  Note that when generating a GUI component state management table, a determination method for adding / deleting a record of a set of attributes (values) and images that control two or more visually different states related to a certain GUI component to the GUI component state management table is as follows: FIG. 3 shows the relationship between the attribute of the GUI part XML tree and the image of the GUI part acquired from the coordinate position information in the GUI part XML tree among the attributes (values) in the GUI part XML tree. And the property value is the same or different), add the corresponding record to the table, × (when the property value is the same and the image is different) delete the record from the table, △ (when the image is the same and the property value is different) ) Indicates that the record is left in the table.) That satisfies the rule shown in FIG. That the number of data related to GUI components that satisfy the rule) can be carried out by selecting the.

  When an operation support request is input from the user, the operation information acquisition unit 7 acquires image information and GUI component configuration information of the operation screen at that time (current).

  The assumed screen generation unit 8 generates an assumed screen from the current operation screen and the state of the GUI component on the current operation screen based on the GUI component state management table in the GUI component state management table storage unit 2. In detail, for example, among the GUI parts on the current operation screen, an image of each state is displayed so as to be selectable by the user for GUI parts that can take two or more visually different states, and the GUI parts on the current operation screen are displayed. An assumed screen in which the visual state is changed to the visual state selected by the user is generated.

  The GUI component XML tree generation unit 9 generates a GUI component XML tree corresponding to the assumed screen. The search execution unit 10 generates a search key from the GUI part XML tree, and searches the operation case storage unit 1 using the search key, thereby searching for operation case information including an operation screen similar to the assumed screen. As the search key described above, for example, it is possible to generate a search key based on text data in which the attribute value is changed for a GUI component whose visual state has been changed, and the attribute value is undefined for other GUI components. it can.

  The GUI part XML tree generation unit 9 and the search execution unit 10 constitute a search unit referred to in the claims.

  The search result display unit 11 displays any one of the time-series operation screens in the searched operation case information so as to be selectable by the user. In addition, when a plurality of operation case information is searched, any one of the operation case information is displayed to be selectable by the user, and any one of the time-series operation screens in the selected operation case information is displayed. Display selectable by the user.

  The collation processing / result display unit 12 collates the operation screen in the operation case information selected by the user with the current operation screen in units of GUI parts, and displays a screen that clearly shows the mismatched portion.

  FIG. 4 shows an operation flowchart in the user operation support apparatus of the present invention. Hereinafter, processing in the user operation support method of the present invention will be described.

  First, the log data acquisition unit 3 executes an automatic test regarding various operation cases in the operation target application, and at that time, acquires image information of a plurality of time-series operation screens generated and GUI part XML trees as log data. The data import unit 4 stores the image information of the operation screen and the GUI part XML tree in the log data in the image image storage unit 1a and the GUI part XML tree storage unit 1b of the operation case storage unit 1, respectively (s1).

  Next, the relation table generation unit 5 generates a relation table indicating the relation between the image information on the operation screen and the GUI part XML tree, and accumulates the relation table in the relation table accumulation unit 1c (s2).

  Further, the GUI component state management table generation unit 6 controls the image of the GUI component and the state of the GUI component that can take two or more visually (image-like) states based on the information in the operation example storage unit 1. A GUI component state management table for managing attribute values to be generated is generated and stored in the GUI component state management table storage unit 2 (s3).

  The processing so far is assumed to be executed before the user performs an operation on the operation target application through the GUI.

  FIG. 5 shows a flow from an operation support request to search key generation.

  When an operation support request is input from the user during operation of the operation target application through the GUI (s4), the operation information acquisition unit 7 acquires image information and GUI component configuration information of the operation screen at that time (current). (S5). In the example of FIG. 5, an apparatus setting screen including GUI parts “Button”, “Slider”, “CheckBox” and the like and a design result screen including GUI part “StatusView” are acquired as operation screens (1).

  FIG. 6 shows an example of a screen at the time of an operation support request, and when the user displays a tab by left clicking the mouse, a “Log search” which is a command corresponding to the operation support request of the present invention is displayed (1- 1). Here, when the user selects “Log search” with the mouse, the processing of (1-3) and (1-4) is performed, and the screen is changed to the search screen window (1-2). In the process (1-3), the image image file on the foreground of the desktop screen is acquired, and in the process (1-4), GUI component configuration information that configures the window left-clicked with the mouse is acquired.

  When the image information and the GUI component configuration information of the current operation screen are acquired, the assumed screen generation unit 8 generates an assumed screen based on the GUI component state management table in the GUI component state management table storage unit 2 (s6). ). In the example of FIG. 5, a GUI component “StatusView” whose attribute value is “Red” included in the design result screen is displayed with a list of images of “StatusView” whose attribute values are “Green” and “Yellow”. (2) Among them, for example, when an image of “StatusView” whose attribute value is “Green” is selected by the user (3), the attribute value of the GUI component “StatusView” whose attribute value is “Red” The image of the “Green” GUI component “StatusView” is replaced and synthesized (4) to generate an assumed image (5).

  FIG. 7 shows an example of the assumed screen generation. Here, the case where the content of the image and the associated text changes in accordance with the change of the attribute value of the GUI part “Slider”, and the attribute of the GUI part “Button” The case where the image changes as the value changes is shown.

  When the assumed screen is generated, the GUI component XML tree generating unit 9 generates a GUI component XML tree corresponding to the assumed screen (6) (s7). The search execution unit 10 generates a search key from the GUI part XML tree (7), and searches the operation case storage unit 1 with the search key to obtain operation case information including an operation screen similar to the assumed screen. Search (s8).

  The search result display unit 11 displays the searched operation case information, but when a plurality of operation case information is searched, any one of the operation case information is displayed so as to be selectable by the user (s9).

  FIG. 8 shows an example of a search result display screen, in which a plurality of operation case information is displayed in units of titles, thumbnails of images that match the search conditions, and comparison and collation buttons. At this time, when the title (an anchor) is clicked, the screen transitions to an operation log time series display screen in which any one of the time series operation screens in the operation case information is displayed to be selectable by the user. In addition, when the comparison / collation button is pressed on the search result display screen, the screen changes to the comparison / collation screen.

  FIG. 9 shows an example of a time-series display screen, which displays a plurality of time-series operation images before or after that on the basis of images displayed as thumbnails on the search result display screen so that the user can select them. When the user selects any of the operation images, the search result display screen returns.

  When the comparison / collation button is pressed on the search result display screen, the collation processing / result display unit 12 collates the operation screen in the operation case information selected by the user with the current operation screen in units of GUI parts, and does not match. A screen specifying the location is displayed (s10).

  FIG. 10 shows the flow of the collation process. A GUI part XML tree of the current operation screen is generated (1), and a property for controlling the state of the GUI part image in the GUI part XML tree of the operation example screen. The values are compared by referring to the GUI component state management table in the GUI component state management table storage unit 2 (2), and the collation result (frame indicating the unmatched part) and the screen are combined and displayed (3).

  FIG. 11 shows an example of a collation result display screen. In the figure, a frame indicated by a broken line indicates a collation result (mismatched portion).

  1: operation example storage unit, 2: GUI component state management table storage unit, 3: log data acquisition unit, 4: data import unit, 5: related table generation unit, 6: GUI component state management table generation unit, 7: operation Information acquisition unit, 8: assumption screen generation unit, 9: GUI part XML tree generation unit, 10: search execution unit, 11: search result display unit, 12: collation processing / result display unit.

“Full-text search system Namazu”, [online], Copyright (C) 2000-2011 Namazu Project. All rights reserved. [October 23, 2012 search] Internet <URL: http://www.namazu.org/ index.html.en> “Flickr”, [online], Copyright (C) 2012 Yahoo! inc. All rights reserved. [Search October 23, 2012], Internet <URL: http://www.filckr.com/> AWM Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, "Content-based image retrieval at the end of the early years," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol.22, no. 12, pp.1349-1380, Dec 2000

Claims (8)

A method for supporting a user operation through a GUI for an operation target application operating on an arbitrary computer,
The arbitrary computer or another computer connected to the arbitrary computer via a network,
Log data of operation cases in the operation target application is acquired, and operation case information including a plurality of time-series operation screens indicated in the operation cases and a GUI part XML tree of the operation screens is stored in the operation case storage unit. Operation case information collection process,
Based on the operation screen in the log data and the GUI component XML tree of the operation screen, a GUI component that manages an image of the GUI component and an attribute value that controls the state of the GUI component that can take two or more visually different states A GUI component state management table generating step for generating a component state management table;
An operation information acquisition step of acquiring image information and GUI component configuration information of the current operation screen when an operation support request is input from the user;
Based on the GUI component state management table, an assumed screen generating step for generating an assumed screen from the current operation screen and the state of the GUI component on the current operation screen;
A search step for searching operation case information including an operation screen similar to the assumed screen from the operation case storage unit;
A search result display step for displaying any one of the time-series operation screens in the searched operation case information so as to be selectable by the user,
The operation screen in the operation case information selected by the user is collated with the current operation screen in units of GUI parts, and a collation process and a result display step for displaying a screen clearly showing the inconsistent portion are executed. User operation support method. Coordinate position information in the GUI part XML tree related to a certain GUI part regarding addition / deletion to the GUI part state management table of a record of a set of attribute values and images that control two or more visually different states related to a certain GUI part If the image and the attribute value are the same or different from each other, the corresponding record is added to the table. If the attribute value is the same and the image is different, the record is added from the table. 2. The user operation support method according to claim 1, further comprising: a GUI component state management table generation step that deletes and stores the record in the table when the image is the same and the attribute value is different. Assumed screen generation process
A step of displaying an image of each state selectable by the user for GUI parts that can take two or more visually different states among the GUI parts on the current operation screen;
The user operation support method according to claim 1, further comprising: generating an assumption screen in which a visual state of the GUI component on the current operation screen is changed to a visual state selected by the user. The search process
A process of generating a GUI part XML tree corresponding to an assumed screen, generating a search key with an attribute value after change for a GUI part whose visual state has been changed, and an attribute value being undefined for other GUI parts When,
The user operation support method according to claim 1, further comprising: searching operation case information including an operation screen similar to the assumed screen by searching the operation case storage unit with the search key. . An apparatus that supports user operation through a GUI for an operation target application that runs on an arbitrary computer,
An operation case storage unit that stores various types of information about operation cases in the operation target application;
A GUI component state management table storage unit that stores a GUI component state management table that manages an image of the GUI component and an attribute value that controls the state of the GUI component that can take two or more visually different states;
Acquire log data of operation cases in the operation target application, and store the operation case information including a plurality of time-series operation screens shown in the operation cases and a GUI part XML tree of the operation screens in the operation case storage unit. An operation case information collection unit to
A GUI component that manages an image of the GUI component and an attribute value that controls the state of the GUI component that can take two or more visually different states based on the image information of the operation screen and the GUI component XML tree in the log data A GUI component state management table generation unit that generates a state management table and stores the state management table in the GUI component state management table storage unit;
An operation information acquisition unit that acquires image information and GUI component configuration information of the current operation screen when an operation support request is input from the user;
Based on the GUI component state management table in the GUI component state management table storage unit, an assumed screen generation unit that generates an assumed screen from the current operation screen and the state of the GUI component on the current operation screen;
A search unit for searching operation case information including an operation screen similar to the assumed screen from the operation case storage unit;
A search result display unit that displays any of the time-series operation screens in the searched operation case information so that the user can select the operation results; and
A user operation support apparatus comprising: a collation processing / result display unit that collates an operation screen selected by a user with a current operation screen in units of GUI parts and displays a screen in which a mismatch portion is clearly indicated. Coordinate position information in the GUI part XML tree related to a certain GUI part regarding addition / deletion to the GUI part state management table of a record of a set of attribute values and images that control two or more visually different states related to a certain GUI part If the image and the attribute value are the same or different from each other, the corresponding record is added to the table. If the attribute value is the same and the image is different, the record is added from the table. 6. The user operation support device according to claim 5, further comprising a GUI component state management table generation unit that deletes and stores the record in the table when the image is the same and the attribute value is different. Among the GUI parts on the current operation screen, the GUI parts that can take two or more visually different states are displayed in a selectable manner by the user, and the visual state of the GUI part on the current operation screen is displayed. The user operation support apparatus according to claim 5, further comprising an assumed screen generation unit that generates an assumed screen that has been changed to a visual state selected by a user. Generate a GUI part XML tree corresponding to the assumed screen, generate a search key with the changed attribute value for the GUI part whose visual state has been changed, and generate an indefinite attribute value for the other GUI part, The user operation support according to claim 5, further comprising: a search unit that searches operation case information including an operation screen similar to the assumed screen by searching the operation case storage unit with the search key. apparatus.

Images