JP6212373B2 - Operation log management apparatus and method - Google Patents

Description

  The present invention relates to a technique for supporting an operation of an operator (user) with respect to a GUI (Graphical User Interface) application program operating on a computer, and more particularly to a technique for managing an operation log provided to an operator as an operation support.

  In recent years, with the advancement of ICT (Information and Communication Technology) technology, business in a company or the like has been systemized, and further, since the business has been consolidated, one operator has to use various business systems. At that time, it is considered that it is possible to provide detailed operation support information that cannot be obtained from the operation manual by searching and comparing similar operations from past operation cases of experts and the like (Non-Patent Documents 1 and 2). reference). As an operation log referred to by the operator as reference information for the operation, it is considered that the operation screen log is suitable for the operator to easily understand intuitively.

  Various methods for retrieving images are considered (see Non-Patent Documents 3 and 4). When the operation screen is converted into a database as a log, a method in which an image search technique is applied to the operation screen image is considered (see Non-Patent Document 5). However, when the image search of the operation screen is performed using the image search technology, the character recognition on the operation screen and the state of the GUI component (button active / inactive, slider position, check box, radio button ON / OFF) There is a problem that a meta search such as a state recognition technique for specifying (off, GUI component position, etc.) cannot be performed. Therefore, a method has been proposed in which information on GUI parts obtained by an Accessibility API (Application Programming Interface) provided from an OS (Operating System) is added as meta information of an operation screen image and the meta information is searched.

Shuji Morisaki, Hayato Kadota, Kenichi Matsumoto, Katsuro Inoue, Koji Torii, “Sharing Software Usage Knowledge Using Function Execution History”, IPSJ Journal, 41 (10), pp.2770-2781, 200010 Moon Yuri Unno, Hikaru Yoshii, Ikuo Hatakeyama, "Knowledge Succession Support System for Server Management Using Operation History", 10th Knowledge Distribution Network Study Group of the Japanese Society for Artificial Intelligence, March 2013 Miki Haseyama, “Current Status of Image / Video Semantic Understanding and Search Interface”, IEICE Journal, Vol.93, No.9, 2010 Yeh, T., Chang, T.-H. and Miller, R.C.Sikuli, "Using GUI Screenshots for Search and Automation", UIST 2009, 183-192 Chang, T., Yeh, T., and Miller, M., "Associating the Visual Representation of User Interfaces with their Internal Structures and Metadata", UIST, 2011, 245-256

  In order to effectively use the operation log such as operation support to the operator, it is important how to search for necessary items from the accumulated operation log. In recent years, WEB applications having advanced graphic functions such as Google Map of Google (registered trademark) have become widespread. In such applications, the operation screen is greatly changed by the operation of the operator. When the operation log is searched by comparing the above, it is difficult to obtain an appropriate operation log. In addition, in a WEB application, a script, an applet, and the like are often operated in an effective environment of various scripts provided by the browser and an application execution environment (for example, JAVA (registered trademark) effective environment) constructed on the browser. Therefore, in the conventional technique, the GUI meta information that can be acquired is not sufficient for the search of the operation log, and it is difficult to search for an appropriate operation log.

  Furthermore, in the search of the operation log, there is a demand not only to search based on the image of the operation screen itself or the meta information but also to compare information (model data) handled by the application itself. For example, in the above-mentioned GoogleMaps, there has been a demand for extracting operation logs based on address information and latitude / longitude information. However, in the above-described conventional technology, it is difficult to satisfy such a request because the search process is performed only with the operation screen image itself or only the meta information.

  The present invention has been made in view of the above circumstances, and one of its purposes is that data that is processed by an application that could not be realized by conventional processing techniques such as GUI meta information and image information. It is to enable analysis of operation logs with high abstraction considering information. Another object of the present invention is to analyze an operation log for an application having an advanced graphic function for handling maps, network topology information, and the like.

In order to achieve the above object, the present invention provides an operation log management apparatus for managing an operation log of a GUI (Graphical User Interface) application operating on a computer, wherein the GUI application includes at least a screen display control unit and data supply. The screen display control unit controls screen display based on the model data acquired from the data supply unit, and the operation log management apparatus acquires GUI meta information related to the operation screen of the GUI application. GUI meta information acquisition unit that performs operation, a model data acquisition unit that acquires model data used in the display of the operation screen, and an operation that records the acquired GUI meta information and model data in a predetermined storage unit in association with each other a log recording unit, the first model data before obtained by the model data acquiring unit By evaluating the similarity between the second model data stored in the storage unit, characterized in that an operation log search unit that extracts the operation logs from the storage unit. According to the present invention, an operation log having an operation screen that matches or is similar to the operation screen can be extracted from the operation screen of the currently displayed GUI application.

Further , according to the present invention, since the model data to be processed by the GUI application and the GUI meta information of the operation screen are recorded as the operation log in association with each other, not only the search using the GUI meta information but also the application Since the search using the information to be processed becomes possible, flexible and sophisticated operation logs can be used. In addition, even an application having an advanced graphic function can be searched not by the operation screen image but by model data used for displaying the operation screen image, so that the operation log can be searched with high accuracy and low load. It can be carried out.

  As an example of a preferred aspect of the present invention, there is further a match point and a mismatch between the first model data acquired by the model data acquisition unit and the second model data related to the operation log extracted by the operation log search unit. The extracted operation log includes a model data comparison unit that compares points for each data element of each model data, and meta information of GUI parts corresponding to the data elements that match and the data elements that do not match in the model data comparison unit. Examples include a GUI component extraction unit that extracts each of the GUI meta information. According to the present invention, a matching point and a mismatching point between two operation screens can be recognized for each GUI component.

  As an example of a preferred aspect of the present invention, an operation screen related to the first model data and an operation screen of the second model data are further based on the meta information of the GUI part extracted by the GUI part extraction unit. Examples include a comparison screen generation unit that generates a comparison screen to be compared. According to the present invention, since the coincidence point and the disagreement point of the two operation screens can be displayed as the comparison screen, it is convenient for the operator.

  In the present invention, as GUI meta information acquisition means, a method using an accessibility API (Application Programming Interface) provided in a computer environment in which a GUI application is operating, or a computer environment in which a GUI application is operating is provided. For example, a method of acquiring GUI meta information by using a DOM (Document Object Model).

  In the present invention, there are various methods for associating GUI meta information with model data. As an example of a preferred aspect of the present invention, the GUI application uses a model data identifier that uniquely identifies each data element of model data acquired from a data supply unit as meta information of a GUI component that displays the model data. Examples include an identifier adding unit to be added. According to the present invention, the operation log recording unit may associate the GUI meta information with the model data using the model data identifier.

  As described above in detail, according to the present invention, as the operation log, the model data that is processed by the GUI application and the GUI meta information of the operation screen are recorded in association with each other, so only the search using the GUI meta information is performed. In addition, since it is possible to search using information that is processed by the application, it is possible to use a flexible and sophisticated operation log. In addition, even an application having an advanced graphic function can be searched not by the operation screen image but by model data used for displaying the operation screen image, so that the operation log can be searched with high accuracy and low load. It can be carried out.

System configuration diagram of the operation log management device The figure explaining the operation of the WEB application Diagram explaining the data structure of model data The figure explaining the data structure of GUI meta information Flow chart explaining operation of operation log management device The figure which shows an example of the operation log table The figure which shows an example of the conversion table which described the correspondence of the data element of model data, and GUI component information Example of operation screen and model data recorded in operation log Example of current operation screen and model data An example explaining model data comparison / collation processing The figure explaining an example of a comparison screen The figure explaining other examples of a comparison screen The figure explaining other examples of a comparison screen

  An operation log management apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a system configuration diagram of an operation log management apparatus.

  As shown in FIG. 1, an operation log management apparatus 100 according to the present embodiment is mounted on a computer terminal 10 operated by an operator. On the terminal 10, an application having a GUI is operating. The operator operates the application using the input device 11, and the operation screen is output to the output device 12. Examples of the input device 11 include a keyboard, a mouse, a trackball, a touch screen, and a microphone for voice input. Examples of the output device 12 include a liquid crystal display. The operation log management apparatus 100 records an operation log of an application operated by an operator, and extracts and displays an operation screen that is identical or similar to the current operation screen of the application operated by the operator from the operation log. The main purpose. The operation log management apparatus 100 according to the present embodiment is also intended to clearly indicate the difference / matching point between the current operation screen and the operation screen extracted from the operation log.

  First, an application targeted by the operation log management apparatus 100 according to the present embodiment will be described. The application according to the present embodiment is a so-called “WEB application” that operates on the WEB browser 200. The application operates by acquiring an application document 210 from the server 260 via the network 250 such as the Internet or an intranet, and interpreting the document with a WEB browser. The WEB browser 200 includes an effective environment of a predetermined script language such as JAVA Script, and realizes an application-specific processing function by executing a script included in the application document 210.

  The application according to the present embodiment includes an application document 210 processed on the WEB browser 200 and various types of data (hereinafter referred to as “model data”) to be processed by the application in addition to the application document 210 with respect to the WEB browser 200. ) To supply a server 260. On the WEB browser 200, a screen display control unit 220 configured to display the model data acquired from the server 260 in a specified manner at a specified position in the operation screen is configured. Further, on the WEB browser 200, a model data identifier assigning unit 230 that assigns a model data identifier described later to GUI meta information derived from the operation screen is configured. The screen display control unit 220 and the model data identifier adding unit 230 are configured by scripts described in the application document 210 as described above. Note that it is preferable to configure the model data identifier assigning unit 230 as a plug-in to the WEB browser 200 because the present invention can be easily applied to an existing WEB application.

  An example of an application according to the present embodiment will be described with reference to FIG. The application shown in FIG. 2 is an example of a map application. When a place name, an address, or the like is input into a text box 501 on the operation screen 500, a store (for example, a restaurant) located around the place indicated by the place name is searched. The marker 510 indicating the position of each store is displayed on the map already displayed on the operation screen 500 as a search result. In such an application, when a place name or the like is entered in the text box 501 and a search button is pressed, a search function unit (not shown) configured by the application document 210 converts the place name into latitude and longitude, and the latitude and longitude are A query 550 as a search key is issued to the server 260 (step S101). The server 260 searches the database (not shown) for store information included around the position indicated by the latitude and longitude included in the query 550 (step S102), and returns the model data 560 as a search result to the WEB browser 200. (Step S103). Based on the acquired model data 560, the screen display control unit 220 controls to display a store or the like corresponding to the data element of the model data 560 on the screen as the marker 510 (step S104). Further, the model data identifier assigning unit 230 assigns a model data identifier that uniquely identifies each data element to the meta information of the marker 510 that is a GUI component related to the display of each data element of the model data 560 (step S105). ).

  In the application according to the present embodiment, various description forms of the model data 560 can be used, and examples thereof include XML (Extensible Markup Language) and JSON (JavaScript Object Notation). FIG. 3 shows an example of model data 560 used in the map application by XML. As shown in FIG. 3, the model data 560 can include a plurality of data elements, and each data element is expressed in a “path” format by expressing the data elements in a tree structure as shown in the lower part of FIG. 3. It can be uniquely identified. The upper diagram in FIG. 3 has a single hierarchical structure, while the lower diagram in FIG. 3 has a multi-hierarchical tree structure, and the two are not consistent. However, this is merely a matter of convenience for explanation. Please be careful.

  In the application according to the present embodiment, various description forms of GUI meta information can be used. For example, those that can be obtained by an accessibility API (Application Programming Interface) provided by the OS, or DOM (Document (Object Model). FIG. 4 shows an example of GUI meta information. The GUI meta information can include meta information of a plurality of GUI parts, and each GUI part can be uniquely identified in the form of “path” by expressing each GUI part in a tree structure. . The meta information of each GUI part includes various kinds of meta information. When the data element of the model data 560 is displayed for the GUI part, the model data identifier assigning unit 230 causes the data to be displayed. A model data identifier that uniquely identifies the element is assigned. Various model data identifiers can be used. For example, an identifier that uniquely identifies the data element of each model data in accordance with the definition of the REST (REpresentational State Transfer) API can be given.

  Next, the operation log management apparatus 100 according to the present embodiment will be described with reference to FIG. 1 again. As shown in FIG. 1, the operation log management apparatus 100 includes a model data acquisition unit 111, a GUI meta information acquisition unit 112, an operation screen information acquisition unit 113, an operation log recording unit 120, and a model data search unit 131. , A model data comparison unit 132, a GUI part extraction unit 133, a display function unit 134, and a storage unit 150 which is a storage unit for accumulating operation logs. The storage unit 150 includes an operation log table 151, a model data storage unit 152, a GUI meta information storage unit 153, and a conversion table for deriving data elements included in the model data from GUI components included in the GUI meta information. 154 is stored. Hereinafter, the operation log management apparatus 100 will be described with reference to the flowcharts of FIGS. 1 and 5.

  The model data acquisition unit 111 monitors the communication traffic between the WEB browser 200 and the server 260 and acquires the detected model data when detecting that the WEB browser 200 has acquired the model data from the server 260.

  When the model data acquisition unit 111 acquires model data, the GUI meta information acquisition unit 112 performs GUI display information on the operation screen of the application after the screen display control unit 220 performs screen display control based on the model data. To get. Note that the model data identifier is included in the GUI meta information acquired here by the model data identifier assigning unit 230. Various methods can be used for acquiring GUI meta information, and examples include a method using an accessibility API and a method using DOM.

  The operation screen information acquisition unit 113 is configured to display the operation information of the operator, the operation date and time, and the operation screen after the operation (that is, the screen display control unit 220 based on the model data) when the model data acquisition unit 111 acquires the model data. Various data (operation screen information) such as a screen image of an application operation screen after display control is acquired.

  The operation log recording unit 120 stores the model data acquired by the model data acquisition unit 111, the GUI meta information acquired by the GUI meta information acquisition unit 112, and various data acquired by the operation screen information acquisition unit 113 in association with each other. The data is recorded in the unit 150 (step S201 in FIG. 5). In this embodiment, as shown in FIG. 6, it is assumed that an operation log is held in the form of an operation log table 151, and model data and GUI meta information are associated with each other by an operation log ID. In this embodiment, as shown in FIG. 6, the operation log table 151 does not hold the entity of model data and GUI meta information, but holds only identification information such as a file name. The data is recorded in the data storage unit 152 and the GUI meta information storage unit 153.

  The model data search unit 131 matches the model data acquired by the model data acquisition unit 112 with respect to the operation screen of the current application (referred to as “current model data” for the sake of convenience) in the past that matches or is similar to the current model data. Model data is extracted from the model data storage unit 152 (step S202 in FIG. 5).

  The model data comparison unit 132 compares the current model data with the model data extracted by the model data acquisition unit 131 in units of data elements, and detects matching points and mismatching points of both model data. In the comparison of the data elements, match / mismatch is evaluated based on the model data path indicating the data elements of the model data (step S203 in FIG. 5).

  The GUI part extraction unit 133 detects the data element of the model data detected by the model data comparison unit 132 and the coincidence point and the disagreement point of each data element of the current model data, and extracts the GUI part corresponding to each data element. (Step S204 in FIG. 5). Specifically, the GUI meta information corresponding to the model data is first obtained from the GUI meta information storage unit 153 with reference to the operation log table 151. Next, when the model data identifier is included in the meta information of each GUI component included in the GUI meta information, the correspondence relationship between the meta information of the GUI component and the data element of the model data specified by the model data identifier Is added to the conversion table 154. The data element corresponding to the GUI component is derived from the model data identifier. As shown in FIG. 7, the conversion table 154 includes a GUI component path (GUI component path) and a model data element path (model data path). Then, the GUI component extraction unit 133 refers to the conversion table 154 and corresponds to the meta information of the GUI component corresponding to the matching data element detected by the model data comparison unit 132 and the mismatched data element. Meta information of GUI parts is extracted.

  The display function unit 134 acquires operation screen information related to the model data searched by the model data search unit 131 from the operation log table 151 of the storage unit 150, and displays the operation screen as a search result as a comparison screen (FIG. 5 step S204). Further, the display function unit 134 performs screen control for showing the coincidence / non-coincidence point obtained by the GUI component extraction unit 133 in an easy-to-understand manner to the operator. Various forms are conceivable as the screen control. For example, (a) highlighting a GUI component that matches the operation screen of the current application, (b) display a GUI component that does not match the operation screen of the current application on the current operation screen. Highlighting existing ones, and (c) newly displaying non-matching GUI parts that do not exist on the current operation screen with respect to the current application operation screen. Can be mentioned.

  An example of operation log search and display by the operation log management apparatus 100 according to the present invention will be described with reference to FIGS. First, it is assumed that an operation log related to the operation screen 800 on the left side of FIG. Here, it is assumed that the data elements 811 to 814 of the model data 810 of the application correspond to the markers 801 to 804 that are GUI parts displayed on the operation screen 800. On the other hand, it is assumed that the operation screen 900 of the application currently operated by the operator is shown on the left side of FIG.

  The operation log management apparatus 100 acquires model data 910 shown on the right side of FIG. Data elements 911 to 913 of the model data 910 correspond to markers 901 to 903 which are GUI parts displayed on the operation screen 900. The model data search unit 131 extracts the model data 810 on the right in FIG. 8 as past model data that matches or is similar to the model data 910. Next, as shown in FIG. 10, the model data comparison unit 132 compares the model elements of the model data 810 and the model data 910 and detects matching model elements and mismatching model elements. Next, the GUI part extraction unit 133 extracts meta information of a GUI part (here, a marker) corresponding to each data element of the model data 810 and the model data 910. Next, the display function unit 134 displays, as a comparison screen, the operation screen 800 on the left in FIG. 9 that is an operation screen related to the model data 810 that is the search result. In addition, the display function unit 134 displays matching points and mismatching points with the past operation screen 800 in various forms on the current operation screen 900. For example, as shown in FIG. 11, elements that are included in the current operation screen 900 but are not included in the past operation screen 800 can be displayed in a different form from other GUI components. Further, for example, as shown in FIG. 12, elements that are not included in the current operation screen 900 but are included in the past operation screen 800 can be displayed in a form different from other GUI components. Further, for example, as shown in FIG. 13, the past operation screen 800 and the current operation screen 900 are displayed side by side, and an arrow or the like is displayed so as to indicate the correspondence relationship for the same element.

  As described above, according to the operation log management apparatus 100 according to the present embodiment, the model data that is processed by the GUI application and the GUI meta information on the operation screen are recorded in association with each other as the operation log. In addition to searching using a search, it is possible to search using information that is processed by an application, so that a flexible and sophisticated operation log can be used. In addition, even an application having an advanced graphic function can be searched not by the operation screen image but by model data used for displaying the operation screen image, so that the operation log can be searched with high accuracy and low load. It can be carried out.

  Although one embodiment of the present invention has been described in detail above, the present invention is not limited to this. In other words, the data structure and the like described in the above embodiment is an example on implementation, and the present invention can be implemented even with other data structures and the like. Moreover, although the said embodiment demonstrated what operates on the WEB browser 200 as an example of an application, other systems, such as a dedicated client server system, may be sufficient. In the above embodiment, the function unit that operates on the WEB browser 200 and the function unit that supplies the model data to the WEB browser 250 are mounted on different devices. However, the model data can be acquired. If so, they may be mounted on the same device. In the above embodiment, the operation log management apparatus 100 is configured on one apparatus. For example, the storage unit 150 is configured in a database on another apparatus, and each functional unit is distributed on different apparatuses. May be arranged. That is, the form of the system including the operation log management apparatus 100 according to the present invention can be arbitrarily configured by various virtual technologies, various distributed technologies, and the like.

  In the above embodiment, the model data identifier assigning unit 230 implemented on the application side is used as a method for associating the data element in the model data with the meta information of the GUI component. However, depending on the implementation form and the implementation environment, The model data identifier assigning unit 230 may be provided on the operation log management apparatus 100 side. In addition, if the mounting form / mounting environment is such that the correspondence between the data elements in the model data and the meta information of the GUI parts can be obtained without using the model data identifier, the model data identifier assigning unit is necessarily required. Note that this is not the case. In addition, the storage unit of the operation log management apparatus 100 holds a conversion table 154 that describes the correspondence between the data elements in the model data and the meta information of the GUI parts, thereby reducing the burden of overlapping processing. However, the association may be derived for each overlap process.

  DESCRIPTION OF SYMBOLS 100 ... Operation log management apparatus, 111 ... Model data acquisition part, 112 ... GUI meta information acquisition part, 113 ... Operation screen information acquisition part, 120 ... Operation log recording part, 131 ... Model data search part, 132 ... Model data comparison part DESCRIPTION OF SYMBOLS 133 ... GUI component extraction part, 134 ... Display function part, 150 ... Memory | storage part, 151 ... Operation log table, 152 ... Model data memory | storage part, 153 ... GUI meta-information memory | storage part, 154 ... Conversion table, 200 ... WEB browser, 210 ... Application document, 220 ... Screen display control unit, 230 ... Model data identifier assigning unit, 250 ... Network, 260 ... Server

Claims (7)

An operation log management device that manages operation logs of GUI (Graphical User Interface) applications that run on a computer,
The GUI application is divided into at least a screen display control unit and a data supply unit, and the screen display control unit controls screen display based on model data acquired from the data supply unit,
The operation log management apparatus includes a GUI meta information acquisition unit that acquires GUI meta information related to an operation screen of a GUI application, a model data acquisition unit that acquires model data used in the display of the operation screen, and the acquired GUI meta information An operation log recording unit for associating the model data with the model data and recording the operation log in a predetermined storage unit ;
An operation log search unit that extracts an operation log from the storage unit by evaluating the similarity between the first model data acquired by the model data acquisition unit and the second model data recorded in the storage unit; An operation log management apparatus characterized by comprising: A model for comparing the matching points and the mismatching points between the first model data acquired by the model data acquisition unit and the second model data related to the operation log extracted by the operation log search unit for each data element of each model data A data comparison unit;
A GUI component extracting unit for extracting the meta information of the GUI component corresponding to the matched data element and the mismatched data element from the GUI meta information included in the extracted operation log. The operation log management device according to claim 1 . A comparison screen generation unit that generates a comparison screen for comparing the operation screen related to the first model data and the operation screen of the second model data based on the GUI component meta-information extracted by the GUI component extraction unit. The operation log management apparatus according to claim 2 . The GUI meta information acquiring unit, one of claims 1 to 3, characterized in that to obtain the GUI meta information by using the accessibility API that GUI application is provided on a computer environments running (Application Programming Interface) The operation log management device according to claim 1. The GUI meta information obtaining unit, according to claim 1 to 3 or 1 and acquires the GUI meta information by using the DOM the GUI application is provided in a computer environment running (Document Object Model) The operation log management device described in the section. The GUI application includes an identifier adding unit that adds a model data identifier that uniquely identifies each data element of model data acquired from a data supply unit as meta information of a GUI component that displays the model data. The operation log management apparatus according to any one of claims 1 to 5 . An operation log management method for managing an operation log of a GUI (Graphical User Interface) application operating on a computer,
The GUI application is divided into at least a screen display control unit and a data supply unit, and the screen display control unit controls screen display based on model data acquired from the data supply unit,
The operation log management apparatus acquires operation screen information of the GUI application, GUI meta information related to the operation screen, and model data used in display of the operation screen, and the acquired operation screen information, GUI meta information, and model data Is recorded in a predetermined storage unit as an operation log ,
An operation log is extracted from the storage unit by evaluating the similarity between the first model data used in the display of the operation screen and the second model data recorded in the storage unit. Operation log management method.

Images